The inputs tab of your notebook configuration allows you to generate datasets from your devices and data tables. Those datasets can then be used within your notebook file to find deeper insights into your IoT application.
To edit a notebook’s input datasets, click the “Inputs” tab after navigating to one of your application’s notebooks. Note: Upon first creating a notebook, this is also the tab to which you will be redirected as configuring inputs is often the next step in the notebook creation process.
The contents of the files generated by your input configurations can be used as datasets within your Jupyter notebook.
In addition to the extra configuration options for each input type as outlined below, each input also requires a File Name, which is the name of the file as it will be written to disk with the contents of the input’s dataset. These names either must be templates or …
- Must be unique across all inputs for the notebook
- May only contain uppercase letters, lowercase letters, numbers, periods (.), underscores (_) or hyphens (-)
- Should (but are not required to) match the file extension of the input
Each file can be referenced by its file name within the directory defined at the
INPUT_DIR environment variable.
For example, to reference a device data input file called
myData.csv as a variable called
myData, your Python code would resemble the following:
import os import pandas as pd myData = pd.read_csv(os.path.join(os.environ['INPUT_DIR'], 'myData.csv'))
You may view a Python code example matching your current notebook configuration by clicking the “View Code Example” link at the top of the Notebook Inputs page.
To add a dataset to your notebook, click the “Add Input” dropdown and choose the type of input you would like to add. There are four different types of data you can use within a notebook …
Most notebooks will generate insights based at least in part on your application’s device state data: the regular reporting of sensors, registers and location information over the life of a device. A great deal can be learned about your IoT solution as a whole when analyzing these large datasets within a notebook.
First, enter a device query for all the devices whose data you would like to use. The combination of all specifically chosen devices and all devices that match any of your chosen tags will be available within your input data.
By default, your input dataset will include all attribute data from all devices returned in your device query. However, you may choose to return only data from a specific set of attributes. If choosing this option, you must select at least one attribute for which to return data.
Generating this dataset requires defining a time range for the data to include in the input file. For most use cases, you will want to return a fixed duration that falls on a natural time breakpoint (such as 60 minutes, 24 hours or 30 days). You may choose one of these predefined options for quickly building such a dataset from the “Time Range” dropdown menu.
In some cases, you may need a more specific time range, in which case you may choose the “Custom” option. Doing so will present you with a series of inputs for defining your time range’s start and end time. Each end of the range can be defined one of two ways:
- Fixed time: This allows for using a specific date and time in the past at which to start or end your dataset. This can be useful for analyzing how your application is evolving over its entire lifetime. Note: Data returned using a fixed start time is still subject to your account’s data retention limits, and as that start time falls behind the limit, you will need to adjust this value.
- Query time minus: This allows for defining times that are relative - in a number of days, hours, minutes or seconds - before the notebook execution’s query time. This can be useful for week-over-week comparisons and the like.
The result of the query will be written to disk as a comma-separated value (CSV) file with the following columns:
- ID: Values within this column correspond to the alphanumeric IDs of the devices returned within your query.
- Timestamp: These values are numbers corresponding to the Unix timestamp (in milliseconds) at which the device ID in the previous column reported state data.
- [ATTRIBUTE]: For each attribute requested in the query — or, if all attributes were requested, each attribute that had at least one state report of its value in the selected time range — a column will exist matching the attribute name. The types of values will vary depending on the attribute type.
Each row in the file corresponds to a single state report from a single device, so the combination of ID and Timestamp per row will be unique within the file. Any attribute column for which a value was reported under that device / timestamp combo will have a value in that row; otherwise the cell will be blank.
A device metadata input returns only configuration information about the devices within your application. No state data is included in the dataset.
In most cases you will create a device metadata input whose device query matches a corresponding device data input; then, you can create a mapping of the IDs from the device data input to their human-readable properties within the metadata input.
Note: The values returned in this query will be for the device configurations as set at the time of import generation. For example, if you set a query time for 30 days in the past, the result of the metadata query will not be the device configuration as it existed 30 days ago, but as it exists now.
Selecting devices here behaves the same way as it does for selecting devices in the device data input type.
The device metadata will be written to disk in a CSV format; the file will have the following columns:
- ID: Values within this column correspond to the alphanumeric IDs of the devices returned within your query.
- Name: These are the human-readable names given to the devices corresponding to the IDs in the previous column.
- Description: If a description was provided for a device, it will appear in this column.
- Class: This corresponds to the device type set for each device.
- tag-[TAG_KEY]: For each unique tag across all the devices returned in the query, there will be a column corresponding to that tag. If a device has a value defined for that key, the value will appear in its respective column. Note: If any of your devices has more than one value set for a specific key, there will be additional “tag” column(s) matching the maximum number of values of that tag on any one device. For any device that does not have multiple values for such a tag, the extra column(s) will be blank.
|589de9bca1975a00017b2291||Engine||Diesel backup generator||standalone||5a8d99177e9707000769fec0||589de9bca1975a00017b2295||5c8bcff0618498000639b9d2||true|
|589de9bca1975a00017b2293||Gateway||Handles reporting from all second-floor sensors||edgeCompute||589de9bca1975a00017b229a||589de9bca1975a00017b2297||false|
Each row in the file corresponds to a single device returned in the metadata query.
Data table inputs return rows from one of your application’s data tables. You may only return the contents of one data table per input; however, you may create multiple inputs that query multiple data tables (or even the same data table).
First, choose a data table from which to return rows for use in your notebook. This is required.
You may optionally return a limited set of the rows from your data table by providing a query. (Not providing a query will return all of the table’s rows.) Defining queries is done in the same way as it is when browsing a data table; you may write simple or advanced queries to return the rows you’d like in your input dataset.
The result will be written to disk as a CSV file with the following columns:
- id: This is the unique ID of each row in the data table.
- [COLUMN]: For each column in the table, there will be a corresponding column in the CSV. The format of the data will depend on the column type.
- updatedAt: This is the timestamp in ISO 8601 format at which any value in the row was last updated.
- createdAt: This is the timestamp in ISO 8601 format at which the row was created.
|5c744e0c08c3220006e23ada||Widget Alpha||Device For Bob||Texas||2019-01-15T22:53:31.902Z||2019-01-15T22:50:44.399Z|
Event data inputs return historical data from your application’s events.
You may filter the returned event data by providing an event query, the filtered result will be writen to disk as a CSV with the following columns:
- ID: This is the unique ID of the event.
- Creation Date: This is the timestamp in ISO 8601 format at which an event was created.
- Last Updated: This is the timestamp in ISO 8601 format at which an event was last updated.
- Source ID: This is the unique ID of the event’s creator.
- Source Type: The type of the event’s creator. It will be one of: “flow”, “user”, “device”, “apiToken”, “experienceUser”, or “public”.
- Level: One of four values: “critical”, “error”, “warning”, or “info”.
- State: One of three values: “new”, “acknowledged”, or “resolved”.
- Subject: This is the event’s title.
- Message: This is additional information pertaining to an event, much like a description to accompany the subject.
- Device ID: If the event has an associated device, that device’s unique ID will appear in this column.
- Data: This is an optional JSON object that may contain additional data about the event.
- Update Number: This is the position in an event’s history that an update took place.
- [TAG]: For each event tag, there will be a corresponding column in the CSV.
|ID||Creation Date||Last Updated||Source ID||Source Type||Level||State||Subject||Message||Device ID||Data||Update Number||Customer|
|5c744e0c08c3220006e23adc||2019-12-03T21:35:41.886Z||2019-12-03T21:38:37.921Z||5d65841f1883530006f8b513||flow||error||new||Low Power||Replace battery in device 1||5db74712edeaea00063c0c1c||0||true|
You may also include data from an external URL as a dataset within your notebook. Many users pull data from their application’s files, but the source of the data can come from any public address.
The result will depend on what is found at the provided URL; it could be (for example) a CSV file, an HTML file, JSON data or even an image.
You must also set a query time when executing a notebook via the Notebook: Execute Node, though in that case you may default the value to the time of the workflow payload.
For example, let’s say we set a query time of “Apr 10, 2019 12:00:00”. Given the custom time range defined in the device data example above …
- Start time is the fixed time of “Mar 23, 2019 16:30:55”
- End time is the query time minus 8 hours
Our device data dataset will return all state reports for the given devices and attributes that occurred between Mar 23, 2019 16:30:55 and Apr 10, 2019 04:00:00.
Requesting another execution with a query time of “May 1, 2019 06:00:00” will return state reports between between Mar 23, 2019 16:30:55 and Apr 30, 2019 22:00:00.
After you have configured all of your notebook’s inputs, you may request a sample export of all datasets. This is useful for developing your Jupyter notebook locally and testing out your code before rolling your notebook out for production use.
Soon after submitting, you will receive an email with links for where to download each of your datasets. Generating the sample data can take anywhere from a couple seconds to several minutes, depending on the size of the datasets in each of the inputs.
Templates are supported to dynamically construct input configurations and file names. The following variables are available: