Skip to main content

Archiving

Archiving is a way to automatically back up an application's data and configuration to third-party cloud services on a daily basis. Your application's archiving settings can be found under the "Archiving" link in an application's subnavigation.

Data Archive

Archiving configuration is broken up into "Data Archiving" (device state data, events, and/or data table rows backed up to cloud storage providers as static files) and "Application Archiving" (workflows, experience resources, device recipes, and more backed up to a Git repository).

Note: Archiving is only available in organization-owned applications.

Data Archiving Configuration​

To enable daily backups of your application data to a cloud storage provider, visit the "Data Archiving" tab of the "Archiving" settings page.

Cloud Provider Settings​

First, choose one of three cloud service providers to receive the backups: Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage.

For each storage service provider, you can select a Service Credential to authenticate the request. If selected, you'll be required to provide a Credential Name from a dropdown list of your Service Credentials. Losant strongly recommends storing this sensitive information in a service credential - including migrating any existing archive configuration into a credential.

If a Service Credential is not selected, each storage service provider requires its own connection configuration:

  • Amazon S3: Requires Region, Access Key ID, and Secret Access Key for configuration.
  • Google Cloud Storage: Requires Account Key (JSON) for configuration. You may optionally provide a Project ID, which will override the Project ID associated with the service account key.
  • Microsoft Azure Blob Storage: Requires Account Name and Account Key for configuration.

All three also require a Bucket (or Container for Azure).

All three have the optional field Directory Inside the Bucket (or Directory Inside the Container for Azure), which specifies a directory for archival files to go into. If left unset, the files will be added to the top-level directory.

All archive configuration fields are templatable (except for Credential Method and Credential Name) via your application globals.

Application Archive Templating

Bucket Permissions​

Before setting up your data archiving configuration, make sure that the credential associated with the third-party cloud provider has the correct permissions.

For Amazon S3, set the user's IAM Policy to the following guidelines, such that bucketname is replaced with the actual bucket name:

{
  "Version": "YYYY-MM-DD",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:getBucketAcl"],
      "Resource": ["arn:aws:s3:::bucketname"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::bucketname/*"]
    }
  ]
}

For Google Cloud Services, create a service account with ObjectAdmin permissions, which are the minimal permissions. More information on permissions within Google Cloud Services can be found in Google Cloud's documentation.

For Azure Blob Storage, ensure that the accessing account has at least the role of Storage Blob Data Contributor assigned to it. Read more on role assignment in Azure.

Device Data​

If "Enable Device Data Archiving" is selected, devices matching your Archive Mode will be archived. After enabling and configuring, there is also an "Archive Backfill" button, which will immediately enqueue a job to archive data from your Archive Data Older Than setting and older.

Application Archive Device Data

Device Data Archive Modes​

When specifying the "Archive Modes," the following options are available:

  • Archive all devices: All devices within the application will be selected for data to archive.
  • Archive all EXCEPT these devices...: A blacklist of devices and/or device tags not to archive.
  • Archive ONLY these devices...: A whitelist of devices and/or device tags only to archive.

Archive Data Older Than Option​

By default, Losant archives data from 31 days prior every time the daily archive job runs. This is because device state data can be overwritten or reported with a past timestamp up to 30 days in the past.

You may optionally change this setting to archive data from sooner than 31 days in the past; however, bear in mind that any state data reported with a past timestamp beyond this setting (i.e. data reported with a timestamp of 20 days ago with an "Archive Data Older Than" setting of 10 days) will not be archived.

Note: Changing this setting to an earlier day - such as going from 31 days to 20 days - will immediately trigger a device archive backfill job to ensure data in the period between the old setting and the new setting (days 21 through 30 in this example) are archived.

Device Data Generated CSV​

Each generated CSV file is placed within a directory named by the ISO timestamp of the archive date. The files themselves will be named by the appplication ID, device ID, and the start and end time of the data contained in the file. An example directory and file would be:

losant-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-568bf74a1ff37b0100f5123e-1512259200000-1512345599999.csv

Each device data CSV will have the following columns:

  • ID (for the device ID)
  • Timestamp (where the timestamp will be represented as a Unix timestamp in milliseconds)
  • ISO Date (where the time is represented in human-readable form)
  • A column named for each attribute on your device

The following is an example of a CSV:

"ID","Timestamp","ISO Date","Current","On","Inuse"
"568bf74a1ff37b0100f5123e",1512259242342,"2017-12-03T00:00:42.342Z",21.666666666666668,1,1
"568bf74a1ff37b0100f5123e",1512259302424,"2017-12-03T00:01:42.424Z",21.733333333333334,1,1
"568bf74a1ff37b0100f5123e",1512259362505,"2017-12-03T00:02:42.505Z",21.8,1,1
"568bf74a1ff37b0100f5123e",1512259422586,"2017-12-03T00:03:42.586Z",21.866666666666667,1,1
"568bf74a1ff37b0100f5123e",1512259482697,"2017-12-03T00:04:42.697Z",21.933333333333334,1,1

Blob Data​

If any of your archives devices Blob-type attributes, the archive will contain the original Base64-encoded strings of data within the CSV.

Archives will never contain the Blob signed URLs because they expire.

Device Metadata Generated JSON​

Alongside the CSV files of data for the individual devices, a JSON file will be created containing metadata about any devices archived for the given day. An example directory and file would be:

losant-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-device-metadata.json

This file will be an array of device objects, one for each of the devices included in the archive for this day. It will be in the same format as the device metadata export that can be requested through the user interface.

Events​

If "Enable Events Archiving" is selected, all events will be archived.

Application Archive Events

There is also a "Full Archive" button, which will immediately enqueue a job to archive all events in the current application.

Event Archive Frequency​

Events are archived daily. For each day all the Events that have been created or modified the day before, Losant will create a directory in the configured storage service that represents the archive date. The directory will contain a CSV containing all events and updates.

Events Generated CSV​

Each generated CSV file is placed within a directory named by the ISO Date of the archive day. The files themselves will be named by the application ID, the word "events", and the archive time of the event data contained in the file. An example directory and file would be:

losant-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-events-1512345599999.csv

The events CSV will have the following columns:

  • ID (for the event ID)
  • Creation Date (ISO Date)
  • Last Updated (ISO Date)
  • Source ID (where the event originated)
  • Source Type
  • Level
  • State
  • Subject
  • Message
  • Device ID
  • Data (in the form of JSON)
  • Update Number (0 for original/parent event)

The following is an example of a CSV:

"ID","Creation Date","Last Updated","Source ID","Source Type","Level","State","Subject","Message","Device ID","Data","Update Number"
"5d24b2fc84cd730006e40cbe","2019-07-09T15:30:04.040Z","2019-07-09T15:30:04.042Z","5b031db82f4fa100050fdf35","user","critical","new","broken device",,,,0
"5d24b2e62f909b0006e2a145","2019-07-09T15:29:42.612Z","2019-07-09T15:29:42.619Z","5b031db82f4fa100050fdf35","user","critical","new","lost device",,,,0
"5c54cbfd10f81d000bac06aa","2019-02-01T22:45:17.629Z","2019-05-08T16:24:57.204Z","5c54c7ff1ef216000626fc38","flow","info","acknowledged","Checking","this is an event",,,0
"5c54cbfd10f81d000bac06aa","2019-05-08T14:57:24.085Z",,"5b0db05e3a4350000933f993","user",,"acknowledged",,"this is an update",,,1

Data Tables​

If "Enable Data Tables Archiving" is selected all data tables matching your "Archive Mode" will be archived.

Application Archive Tables

Data Tables Frequency​

Data tables are archived daily. Each day, Losant will create a directory in the configured storage service that represents the archive date. The directory will contain a CSV for each data table.

Data Tables Archive Modes​

When specifying the "Archive Mode", the following options are available:

  • Archive all data tables: All data tables within the application will be archived.
  • Archive all EXCEPT these data tables...: A blacklist of data tables not to archive.
  • Archive ONLY these data tables...: A whitelist of data tables only to archive.

There is also a "Full Archive" button which will immediately enqueue a job to archive all Data Tables connected to the application.

Data Tables Generated CSV​

Each generated CSV file is placed within a directory provided or named by the ISO timestamp of the archive date. The files themselves will be named by the application ID, the string "data-table-data", and the data table ID. An example directory and file would be:

losant-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-data-table-data-5d24b2fc84cd730006e40cba.csv

Or if you provide a directory in the configuration:

losant-bucket/2017-12-03T00:00:00.000Z/your-directory/568beedeb436ab01007be53d-data-table-data-5d24b2fc84cd730006e40cbt.csv

Each data table CSV will have the following columns:

The following is an example of a CSV:

"id","Hour","PriceScale","updatedAt","createdAt"
"5d2d45183c4fad0006e657b0",0,0.8199146941389697,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b1",1,0.7345838500288444,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b2",2,0.5752394968861998,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b3",3,0.44334167226038557,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"

Application Archiving Configuration​

To enable daily backups of your application configuration to a Git repository, visit the "Application Archiving" tab of the "Archiving" settings page and fill out the form.

Application Archive

Git Repository Settings​

Enter the following to configure the daily backup:

  • Credential: Select one of your application's Git service credentials for authenticating the export request.
  • Branch Name: Optionally, provide a branch name to commit the resources to. If not provided, this defaults to the repository's default branch.
  • Directory in Repository: Optionally, provide a directory within the repository in which to push resources. This defaults to the root of the repository if not provided.
  • Commit Message: Optionally, provide a commit message as part of the Git commit for the daily backup.

Unlike data archiving configuration, these values do not support string templates for referencing values from application globals.

Select Resources​

Most application resources and their configuration are automatically included in the daily backup. You may opt in to including the following resources as well:

  • Devices: Selecting this option will include all of your device configuration in the daily export. This does not include any device data nor any access keys tied to your devices.
  • Data Table Rows: Selecting this option will include all of the rows stored within your Data Tables in the export. If unchecked, your export will still include all of your tables and their column configurations, but with no content in the tables. (Data table rows can also be archived through the Data Archiving options.)
  • Files: Selecting this option will include all of your files in the daily export.

Resources That Will Never Be Archived​

The following application resources are never included in the application archive:

Requesting an Archive Run​

If you have defined either archive configuration, you will see a "Request Archive Now" button at the top of the respective form. Pressing it will enqueue an archive job, backing up your application's data or configuration immediately.

Deleting Archive Configuration​

To delete either your data archive application archive configuration, press the "Delete Archive Settings" button at the bottom of the respective form. Doing so takes effect immediately, and any future daily backups will not run until settings are restored.

Was this page helpful?

Still looking for help? You can also search the Losant Forums or submit your question there.