TARGETS

In order to compile a Spatial application for a particular target, you must follow these general steps:

  1. Compile the Spatial application to generate target-specific code, such as Scala, C++, or Chisel

  2. Compile the target-specific code

  3. (Optional) Export and unpack the compiled code to the target platform

  4. Run the compiled code

1) bin/spatial <app name> <--help or options>
  
2) cd <generated dir> && make

3) x <generated tar>
  
4) bash run.sh <args>

Note that if the generated directory already exists, you will receive a warning. Not deleting the generated directory before step 1 could result in unexpected errors when executing step 2.

Select a target below to see specific instructions for how to compile for that particular device.

+ SCALA SIMULATION

Notes

The Scala backend is a useful tool for debugging at the algorithm level. It has a few advantages and disadvantages, however:

• Advantages:
      o println statements in the Accel will actually print while the app simulates
      o Fastest way to go from source code to execution
• Disadvantages
      o Not cycle accurate.
      o Will not expose pipeline or parallelization bugs
      o Cannot target any devices from generated code.
      o Execution can be slow

Process

      $ # Compile app in Spatial
      $ bin/spatial --sim<br>&nbsp;&nbsp;&nbsp;&nbsp;  $ # Compile backend
      $ cd gen/<app name> && make
      $ # Run app
      $ bash run.sh "<args>"

Note that you can use the input arg -h to see help about the arguments for the app

+ VCS SIMULATION

Notes

VCS is a powerful simulator by Synopsys. The VCS backend is a useful tool for testing the application as it would run on a real FPGA. It has a few advantages and disadvantages, however:

• Advantages:
      o Cycle accurate simulation
      o Debuggable waveforms (see below to learn how to enable logging)
• Disadvantages
      o Requires VCS license.
      o Simulates relatively slowly
      o Does not simulate true interface to peripherals on real FPGAs.

Process

      $ # Compile app in Spatial
      $ bin/spatial <app name> --synth --fpga=VCS <other options>
      $ # Compile backend
      $ cd gen/<app name> && make
      $ # Run app
      $ bash run.sh <args>

Note that you can use the input arg -h to see help about the arguments for the app

If you would like to enable waveform debugging, either with vcd or vpd, edit the file gen/<app name>/vcs.hw-resources/Top-harness.sv and turn either reg vcdon or reg vpdon to 1.

+ AMAZON EC2 F1

Notes

Vivado and a license to run on the desired Amazon FPGA. We tested this tutorial both on Amazon’s FPGA Developer AMI which contains all the required software tools, as well as locally by following these instructions. Clone Amazon’s EC2 FPGA Hardware and Software Development Kit to any location:

      $ git clone https://github.com/aws/aws-fpga.git

Spatial was most recently tested with version 1.3.3 of this repository (git commit 934000f9a57c0cde8786441864d5c6e0cf42fef9).
Set the AWS_HOME environment variable to point to the cloned directory. Also source the AWS setup scripts. The HDK script is needed for simulation and synthesis, and the SDK is needed to create the host binary:

      $ export AWS_HOME=/path/to/aws-fpga
      $ cd /path/to/aws-fpga/
      $ source /path/to/aws-fpga/hdk_setup.sh
      $ source /path/to/aws-fpga/sdk_setup.sh

The AWS backend allows you to simulate using XSIM, as well as synthesize and deploy on the actual FPGA.

Process (Simulation)

We intend for Spatial applications to work in hardware without requiring simulation. But for the alpha release of Spatial, the user may prefer to first simulate the design before paying to open an F1 FPGA instance. This tutorial describes how to simulate your Spatial application using the simulation environment provided by Amazon in their Development Kit.
These steps can be done on any machine with the Vivado XSIM simulator. We did this both on a local machine as well as on an EC2 machine with the Amazon FPGA Developer AMI. An FPGA instance is not needed for this simulation tutorial.

      $ # Compile app in Spatial
      $ bin/spatial <app name> --synth --fpga=AWS <other options>
      $ # Compile backend
      $ cd gen/<app name> && make aws-sim
      $ # Run app
      $ ./Top <args>

Note that you can use the input arg -h to see help about the arguments for the app

Process (Hardware)

Unlike simulation, running on the F1 requires a few simple manual steps. These depend on your personal AWS account (EC2 and S3). Specifically, following synthesis Amazon requires the bitstream to be uploaded to your S3 account, and an EC2 account is needed to launch an F1 instance to run the spatial application in hardware.
This tutorial describes the following steps:

  • Authenticating your AWS account
  • Generating and synthesizing a Spatial design. In our experience, synthesis/place/route takes 4-12 hours depending on design size
  • Uploading the bitstream (AKA design checkpoint, or DCP) to Amazon S3 and waiting approximately 1 hour for the Amazon FPGA Image (AFI) associated with this bitstream to become valid
  • Opening an F1 instance through your EC2 account
  • Running the spatial application


Step 1: Authenticating your AWS account

Follow these steps to create a file rootkey.csv. This file can be placed anywhere, and will be needed for later steps to run commands associated with your AWS account.
Then add the path to that file to your .bashrc as follows:
export AWS_CONFIG_FILE=/path/to/rootkey.csv

Step 2: Generate and Synthesize your application

The following will generate the F1 design and run synthesis/place/route to begin creating the Amazon FPGA Image (AFI). We tested this both using Amazon’s FPGA Developer AMI and also locally using their instructions.
      $ # Compile app in Spatial
      $ bin/spatial <app name> --synth --fpga=AWS <other options>
      $ # Compile backend
      $ cd gen/<app name> && make aws-F1-afi
      $ # Run app
      $ sudo ./Top <args>
Notice that once this command completes, Vivado synthesis will be running in the background. A file called create_spatial_AFI_instructions.sh has also been created. We will discuss this file in the next step.
Vivado will have completed once the *.vivado.log file in aws-fpga/hdk/cl/examples//build/scripts has printed Finished creating final tar file in to_aws directory.. You can also check if the vivado process has stopped.

Step 3: Creating the AFI

See the generated file create_spatial_AFI_instructions.sh, which is an executable bash script that uploads the DCP to S3 and then runs the create-fpga-image command.

      $ bash create_spatial_AFI_instructions.sh

Running these commands will require the AWS Command Line Interface. We tested with version 1.11.78.
Once create-fpga-image has been run, wait for the logs directory in S3 to be filled, and ensure that the AFI state in the file called State is “available”.

Step 4: Opening an F1 instance

Start an F1 instance in the AWS console. We tested with Amazon’s FPGA Developer AMI although this might not be necessary.
If you already have an existing F1 instance (e.g. for a previous Spatial application), skip to Step 5. If this is your first time starting the F1 instance, follow the one-time setup steps below. Clone Amazon’s EC2 FPGA Hardware and Software Development Kit to any location, e.g. /home/centos/src/project_data:

      $ git clone https://github.com/aws/aws-fpga.git

Put the following (replacing with your chosen path above) in your .bashrc:

      pushd /home/centos/src/project_data/aws-fpga/
      source /home/centos/src/project_data/aws-fpga/sdk_setup.sh
      popd


Source the .bashrc:

      $ source ~/.bashrc

Then follow these instructions to build and install the required EDMA driver. Summary:

      $ cd sdk/linux_kernel_drivers/edma/
      $ make
      $ echo 'edma' | sudo tee --append /etc/modules-load.d/edma.conf
      $ sudo cp edma-drv.ko /lib/modules/uname -r/
      $ sudo depmod
      $ sudo modprobe edma-drv


Step 5: Running the Spatial application

Generate the host (software) binary using:

      $ make aws-F1-sw

You can do this on your local machine and copy over the binary to the F1 (this might require changing permissions to run it), or compile the binary on the F1 instance. To do it on the F1 instance, you only need the software/runtime and software/include directories of the generated Spatial AWS application (e.g. aws-fpga/hdk/cl/examples/<app name>), and can compile using make all in software/runtime.


Also modify the file load.sh in software/runtime to paste in the agfi ID returned above, if this has not already been done.

Run the application using the commands below in the runtime directory:


      $ bash load.sh
      $ sudo ./Top '`



Note that you can use the input arg -h to see help about the arguments for the app

+ XILINX ZYNQ FAMILY

Notes

To run on this target, you will need a Zynq 7000 board, such as the ZC706 or Zedboard, and a valid license for Vivado for this target. You will also need the appropriate cross-compiler.:

      $ sudo apt-get install libc6-armel-cross libc6-dev-armel-cross binutils-arm-linux-gnueabi libncurses5-dev

The generated executable for this target assumes you have a Linux OS installed on the board and have an executable in your path called prog_fpga. To create this executable, download this file and do the following on the Zynq:

      $ mv prog_fpga /usr/bin
      $ chmod +x /usr/bin/prog_fpga


Process

The following process uses the environment variable CLOCK_FREQ_MHZ to configure the clock frequency, or else it uses 125MHz by default.
      $ # Compile app in Spatial
      $ bin/spatial <app name> --synth --fpga= <other options>
      $ # Compile backend
      $ cd gen/<app name> && make # This may take hours
      $ # Transfer to Zynq
      $ scp *.tar.gz <user>@<ip>:
      $ # ssh into Zynq
      $ ssh <user>@<ip>
      $ # Unpack tar.gz and create proper directory structure
      $ tar -xvf <app name>.tar.gz -C <app name> && cd <app name> && mkdir verilog && mv accel.bit.bin verilog
      $ # Run app
      $ sudo ./Top <args>

Note that you can use the input arg -h to see help about the arguments for the app

When you run the program, you will see it print out a line that looks like Design done, ran for 0.003507 ms. This tells you how long the FPGA was active for.

If you want to change the CPU code, you can make edits in gen/<app name>/cpp. The file TopHost.cpp is contains the main() function and uses header files from both the cpp directory and zynq.sw-resources.

+ XILINX Ultrascale MPSoC

Notes

To run on this target, you will need a ZCU102 board or similar and a valid license for Vivado for this target. You will also need the appropriate cross-compiler.:

      $ sudo sudo apt-get install g++-aarch64-linux-gnu

The generated executable for this target assumes you have a Linux OS installed on the board. We suggest either building Petalinux with the appropriate FPGA manager drivers, or using the prebuilt image here.
You must also have an executable in your path called prog_fpga. To create this executable, download this file and do the following on the ZCU:

      $ mv prog_fpga /usr/bin
      $ chmod +x /usr/bin/prog_fpga


Process

The following process uses the environment variable CLOCK_FREQ_MHZ to configure the clock frequency, or else it uses 125MHz by default.
      $ # Compile app in Spatial
      $ bin/spatial <app name> --synth --fpga=ZCU <other options>
      $ # Compile backend
      $ cd gen/<app name> && make # This may take hours
      $ # Transfer to Zynq
      $ scp *.tar.gz <user>@<ip>:
      $ # ssh into Zynq
      $ ssh <user>@<ip>
      $ # Unpack tar.gz and create proper directory structure
      $ tar -xvf <app name>.tar.gz -C <app name> && cd <app name> && mkdir verilog && mv accel.bit.bin verilog
      $ # Run app
      $ sudo ./Top <args>

Note that you can use the input arg -h to see help about the arguments for the app

When you run the program, you will see it print out a line that looks like Design done, ran for 0.003507 ms. This tells you how long the FPGA was active for.

If you want to change the CPU code, you can make edits in gen/<app name>/cpp. The file TopHost.cpp is contains the main() function and uses header files from both the cpp directory and zcu.sw-resources.

+ Altera Arria10 SoC

Notes

To run on this target, you will need an Arria10 board or similar and a valid license for Quartus for this target. You will also need the appropriate cross-compiler.
The generated executable for this target assumes you have a Linux OS installed on the board, and that the core .rbf contains firmware to interact with DRAM.

Process

The following process uses the environment variable CLOCK_FREQ_MHZ to configure the clock frequency, or else it uses 125MHz by default.
      $ # Compile app in Spatial
      $ bin/spatial <app name> --synth --fpga=Arria10 <other options>
      $ # Compile backend
      $ cd gen/<app name> && make # This may take hours
      $ # Transfer to Zynq
      $ scp *.tar.gz <user>@<ip>:
      $ # ssh into Zynq
      $ ssh <user>@<ip>
      $ # Unpack tar.gz and create proper directory structure
      $ # TBA
      $ # Run app
      $ sudo ./Top <args>

Note that you can use the input arg -h to see help about the arguments for the app

When you run the program, you will see it print out a line that looks like Design done, ran for 0.003507 ms. This tells you how long the FPGA was active for.

If you want to change the CPU code, you can make edits in gen/<app name>/cpp. The file TopHost.cpp is contains the main() function and uses header files from both the cpp directory and zynq.sw-resources.

+ ASIC

Notes

To run on this target, you will need the appropriate Synopsys licenses.
More TBA

Process

The following process uses the environment variable CLOCK_FREQ_MHZ to configure the clock frequency, or else it uses 125MHz by default.
      $ # Compile app in Spatial
      $ bin/spatial <app name> --synth --fpga=ASIC <other options>
      $ # Compile backend
      $ cd gen/<app name> && make # This may take hours
      $ # You should now have reports and generated code for an ASIC, details TBA

+ PLASTICINE

Notes

To run on this target, you will need a Plasticine.
More TBA

Process

      $ # Compile app in Spatial
      $ bin/spatial <app name> <ARGS TBA> <other options>
      $ # Compile backend
      $ cd gen/<app name> && make # This may take hours
      $ # You should now have generated PIR