Data uploaded to Senate can be made available later via the Data Republic Catalog or added to a Project in Senate for exchange.
There are two ways to upload files or folders onto the Senate Platform. Select the method that applies to your file:
- Using SFTP (if your file size is larger than 100MB)
- Using the Senate app (if your file size is less than 100MB); click the 'Add file' button on the Manage Data screen to upload a file.
Optimal file size
Prior to uploading your data to Senate, create a directory (folder) for your files. We recommend removing the header row and breaking up large files into many smaller files (of less than 1GB each when uncompressed) rather than trying to load one large file to Senate; then compress each of these files prior to upload to Senate (e.g. you can use gzip compression)
This will make it easier you to troubleshoot any issues that may occur during the file upload process or during the subsequent loading of a folder or individual file to a table you create.
Recommended file formats
Senate accepts both structured and unstructured data. However, if you will be working with tables, we recommend using csv / tsv or any characters delimited file, either in plain text or gzipped; gzip compression is recommended for faster uploads. You can specify the field separator later when loading data to the table.
Note: Senate does not currently support the parquet file format.
Uploading files via SFTP
If the file you're working with is more than 100MB, you will need to upload your data file via SFTP. You can upload anything: including data files, images, scripts, documents, and so on. Later on in the process, we can create a database table for any data files, and package the files and/or tables for use in projects.
- Use your preferred SFTP client to connect sftp.datarepublic.com.au with your Senate user name and password.
- Leave the files in the organization’s share folder.
- If you will be uploading many files, it may be convenient for you to create a directory (folder) within your organization share folder to upload the files to.
Check that the upload was successful:
- Log into Senate.
- Click Manage Data.
Your file should be listed under your organization’s folder.
Note: No special firewall rules should be required to connect to the SFTP site. However, if you experience issues connecting, you may need to contact your IT department to whitelist the following IP addresses: 22.214.171.124 and 126.96.36.199 (TCP Port 22).
Adding files using the Senate app
If the file you're working with is less than 100MB, you can simply upload your file via the Manage Data screen.
To add a file to your organization folder in Senate:
Navigate to 'Manage Data' from the main menu on the left.
The 'Files' tab should be displayed:
- You should see your organization folder listed. Click on this folder.
- Click 'Add File' and select a file from your computer to upload or you can drag and drop your file.
You will see a 'File Uploaded' message displayed once the file upload is complete.
The file will be visible under your organization folder.
The next step in preparing your data to make it available for the Data Republic Catalog or for a project on Senate, is creating a database.