This article provides details on uploading and managing your files on the Data Republic platform.
Files uploaded to the Data Republic platform can later be packaged and added to a specific project for sharing.
There are two ways to upload files or folders onto the Data Republic Platform. Select the appropriate method for your file size:
- Using SSH File Transfer Protocol a.k.a. SFTP (if your file size is larger than 100MB)
- Using the Data Republic app (if your file size is less than 100MB); click the 'Add file' button on the Manage Data screen to upload a file.
In this article you will learn about:
- Optimal file size
- Recommended file formats
- Uploading files via SFTP
- Adding files using the Data Republic app
- Upload your Code or Algorithms using a Docker container
- Deleting Files
Optimal file size
We recommend removing the header row and breaking up large files into smaller files (of less than 2GB each when uncompressed) rather than trying to load one large file to the Data Republic platform; then compress each of these files prior to upload to the Data Republic platform (e.g. you can use gzip compression).
This will make it easier for you to troubleshoot any issues that may occur during the file upload process or during the subsequent loading of a folder or individual file to a table you create.
Recommended file formats
The Data Republic platform accepts both structured and unstructured data. However, if you will be working with tables, we recommend using csv / tsv or any characters delimited file, either in plain text or gzipped. The Data Republic platform does not currently support the parquet file format.
Note: gzip compression is recommended for faster uploads. If the file type is csv, and the file will be loaded into a table, the extension .csv.gz is recommended.
Uploading files via SFTP
If the file you're working with is larger than 100MB, you will need to upload your data file via SFTP. You can upload anything including data files, images, scripts, documents, and so on. After your files have been uploaded to the Data Republic platform we can create a database table for any structured data files, and then package the files and/or tables for use in projects.
- You can use your preferred SFTP client and connect with your Data Republic user name and password. SFTP servers for each region are listed below:
- Move files from your local directory to the organization’s share folder.
- If you will be uploading many files, it may be convenient for you to create a directory (folder) within your organization share folder to upload the files to.
It's always recommended to check that the upload was successful, you do that by:
- Sign in to the Data Republic.
- Click Manage Data.
- Click the Files tab
If your STFP upload is successful, you will see the file(s) listed in your organization’s folder.
Note: No special firewall rules should be required to connect to the SFTP site. However, if you experience issues connecting, you may need to contact your IT department to whitelist the following IP addresses: 220.127.116.11 and 18.104.22.168 (TCP Port 22).
Uploading files using the Data Republic app
If the file you're working with is less than 100MB, you can simply upload your file via the Files screen in Manage Data.
To add a file to your organization folder in the Data Republic platform:
- Navigate to Manage Data from the main menu on the left.
- Click the Files tab:
- You should see your organization folder listed. Click on this folder.
- You can drag and drop your file or click Add File and select a file from your computer to upload.
5. You will see a File Uploaded message displayed once the file upload is complete and the will be visible under your organization folder.
The next step in preparing your data to make it available for the Data Republic for a Project, is creating a database.
Upload your Code or Algorithms using a Docker container
Data Republic supports running Rootless Docker securely, within CPU Linux Workspaces. Rootless docker works by starting the docker daemon as the current user, as a pose to root in traditional installations.
1. Prepare your Docker container locally and sav as a .tar file (example below)
docker save <image name>:<tag> > <name>.tar
If you have loaded the wrong file to the Data Republic platform, or if you would like to remove a file from the Data Republic platform, you can delete the file by:
1. Navigate to Manage Data from the main menu on the left.
2. Click the Files tab .
3. Click on whichever file you would like to delete.
4. Click Delete.