Jide Oyelayo
CV

Work / 1.1 FPGA video processing

Case study 1.1 · Systems

FPGA video processing

Edge filters with Prewitt and Sobel weights in hand-written Verilog, run on live camera video on a Zybo Z7-20 FPGA; the design does not meet timing.

Date
Apr 2023
Role
Author
Stack
Zybo Z7-20, Verilog, Vivado 2019.1, C++ (Xilinx SDK)

Status: draft. Certainty: Utilisation and timing are read from the Vivado 2019.1 reports committed in the repository in April 2023, not re-run. Filter behaviour comes from an RTL simulation checked against a bit-accurate model, and latency is derived from the code; the simulation is in the repository, added by its pull request #1. Nothing was measured on the board. The capture and display chain is Digilent material; the hand-written part is the filter module.

Summary

This project runs edge filters on live camera video on a Digilent Zybo Z7-20 FPGA board. The video comes from an OV5640 camera (a Digilent Pcam 5C) over MIPI CSI-2 and leaves over HDMI. The filter is hand-written Verilog, inserted into Digilent’s Pcam 5C demonstration design and built in Vivado 2019.1; the software that configures the camera and the video path is Digilent’s C++ demonstration application, unmodified. There are two builds, one with Prewitt weights and one with Sobel weights.

The Vivado reports committed with the project show that neither build meets timing: the slowest path needs about 97–99 ns against a 6.734 ns pixel clock. An RTL simulation shows that the filter is not the textbook Prewitt or Sobel operator either: a line-buffer bug gives it a four-line window instead of a 3×3 one. Nothing was measured on the board, so whether its output matches the simulation is unknown. This page stays a draft.

Context and constraints

Edge detection is normally a software step. Doing it on an FPGA means the filter sits in the video path itself, so each frame is processed as it streams through rather than after it has been stored. That makes the design a hardware question: how the pixel stream is buffered, how the filter kernel is applied to it, and how the result is sent back out as video.

The limits below are read from the block design, the driver code, the filter and the committed reports.

Operating limits
LimitValue
Video input sourceOV5640 camera (Pcam 5C) over MIPI CSI-2, from the block design and driver code
Frame size and rate1920×1080 at 60 Hz on HDMI output; the camera runs at 1080p30 by default, so each camera frame passes through the filter twice. 1280×720 is selectable from the software menu, but there every second output line is white
Pixel clock148.5 MHz at 1080p60 (74.25 MHz at 720p), from the design; timing is not met at 148.5 MHz
Frame format the filter assumes24-bit pixels in {R, B, G} order, one per clock, active-high hsync, lines longer than 1,800 clocks; at 1080p the rightmost 310 columns are always white
Filter latency4,404 clocks (29.66 µs at 1080p60) from the pixel a window is centred on to its display, in the X and Y modes; derived from the code and checked in simulation
Camera-to-HDMI latencynot measured: it includes the frame buffer in DDR and needs the board
Logic and memory useFilter alone: 4,872 LUTs (Prewitt) and 4,845 (Sobel), half of them LUT RAM; no block RAM or DSP slices (table below)

My role

The repository holds two complete Vivado projects built on Digilent’s Zybo Z7-20 Pcam 5C demonstration design. I wrote the video_processing filter module and its square-root helper, 410 lines of Verilog across the two projects, and made the edits that insert the filter into the block design and assign the buttons, switches and LEDs. The camera and display chain is vendor material: the MIPI CSI-2 receiver, Bayer-to-RGB and gamma IP, the VHDL clocking and reset helpers, the OV5640 driver and the C++ application come from Digilent unmodified or are generated by Vivado, and are not claimed here.

SOURCES.md in the repository lists which files are hand-written and which are vendor or generated, and how that was established.

Architecture

The camera feeds a MIPI CSI-2 receiver, Bayer-to-RGB conversion and gamma correction, and the frames are written to DDR memory by a video DMA. They are read back in display timing and pass through the video_processing Verilog module, then out through an HDMI encoder. So the filter sits in the output stream after the frame buffer, not in the capture path.

Inside the module, the pixel is converted to greyscale with a shift-and-add approximation of luma, about 0.297 R + 0.594 G + 0.125 B, so white maps to 253. The {R, B, G} channel order matches the vendor chain. Three 1,800-entry line buffers in LUT RAM were meant to give a 3×3 window for the Prewitt or Sobel kernel. Buttons choose the X gradient, the Y gradient or the magnitude, the square root of X² plus Y²; a switch bypasses the filter and shows the greyscale image. There is no threshold or smoothing stage. The diagram shows the intended stages only; it carries no sizes or timings.

Camera inLine buffersPrewitt / Sobel kernelGradient magnitudeHDMI out
Diagram A. Video processing pipeline Video frames passing through the intended Prewitt and Sobel filter stages on the Zybo Z7-20 FPGA. Schematic. Illustrative data

Each Vivado project has its own copy of the filter. The two copies differ only in kernel weights, register names and one Prewitt-only button line, and share every bug below. The README’s HDMI input is wrong: the block design and application code show the camera input. The hand-written filter is Verilog; the VHDL in the repository is vendor code.

Key decisions

Filter in hardware. The filter is written in Verilog and sits in the video path. The aim was real-time operation on live video; the alternative, filtering in software, would put the cost on the processor. The committed timing reports show that the aim is not met at 148.5 MHz (below). The demonstration videos show the board producing output, but correct operation at that rate is not guaranteed.

Two filters, two projects. Prewitt and Sobel are separate builds. Each is simple to inspect and demonstrate; the cost is duplicated source, so each fix has to be made twice. Whether a single configurable design would have been better is to write.

Build on the vendor demo. The camera, capture, clocking and display chain come from Digilent’s reference design rather than being rewritten. That keeps the effort on the filter, at the price of depending on vendor code.

Evaluation and results

Tool output is read from the Vivado 2019.1 reports committed on 24 April 2023 for part xc7z020clg400-1: the routed timing summary and placed utilisation in */*.runs/impl_1/, and the filter alone, synthesised out of context, in */*.runs/system_video_processing_0_0_synth_1/. Nothing was measured on the board, and the demonstration videos (Prewitt on YouTube, linked rather than embedded) are qualitative only.

Tool outputPrewittSobel
Filter LUTs, of which LUT RAM4,872 (9.16%), 2,4364,845 (9.11%), 2,436
Filter registers, including 16 latches250246
Whole-design LUTs12,616 (23.71%)12,592 (23.67%)
Whole-design block RAM tiles10.5 (7.50%)10.5 (7.50%)
Pixel-clock worst negative slack−92.154 ns−90.287 ns
Failing pixel-clock endpoints3,950 of 25,9043,880 of 25,896

All ten worst pixel-clock paths in each report lie inside the filter. Eight run from the squared-gradient register through the square root to the output: 96–99 ns over 328–330 logic levels, because the square root is a combinational divider inside a latch loop. That is roughly 10 MHz against the 148.5 MHz pixel clock, worked out from the report rather than run. The other two read the line buffers, with −11.1 ns (Sobel) and −13.7 ns (Prewitt) of slack, and every gradient mode uses that path. Two violations on the camera receiver’s input clock, −2.450 ns setup and −1.038 ns hold, are in Digilent’s MIPI receiver, not in the filter. The block RAM belongs to the vendor IP.

RTL simulation. The filter was run in Icarus Verilog against a cycle-accurate software model: 62 of 62 cases bit-identical, 10,291,200 output words, stimulus seed 1, deterministic. Two edits were needed for a simulator to accept the code and give the hardware’s answer, an unnamed instance and a blocking assignment; neither should change the synthesised logic, but that has not been confirmed in Vivado. Against the textbook 3×3 operator on uniform noise, the filter agrees on 0.23–0.56% of pixels, chance level for 8-bit values (0.39%). The cause is a bug: the line buffers’ write pointer advances every pixel instead of every line, so the impulse response has six taps over four lines rather than a 3×3 window. The output is still edge-like, since the taps sum to zero, but it is not the Prewitt or Sobel gradient. The testbench, model, comparison script, recorded output and results write-up are in the repository, added by its pull request #1, which is merged.

Derived from code. The filter adds two lines plus four clocks between the pixel a window is centred on and its display, 4,404 clocks or 29.66 µs at 1080p60; the greyscale bypass adds 3 clocks. Throughput is one pixel per clock with no back-pressure, so 148.5 MHz including blanking, but only if timing were met.

Method what is measured, and how

Question
Does edge detection on live video run in real time on the Zybo Z7-20, in hardware rather than software?
Input
OV5640 camera (Pcam 5C) over MIPI CSI-2; 1080p output at 60 Hz by default
Hardware
Zybo Z7-20 (xc7z020clg400-1) with a Pcam 5C camera and an HDMI display; the display model is not recorded. Simulation on an Apple M1 Pro
Software
Vivado 2019.1 (build 2552052) and Xilinx SDK for the build; Icarus Verilog 13.0 and Python 3.15 for the simulation
Reported
Utilisation and timing from the committed reports; filter output in RTL simulation, 62 cases, exact counts with no spread; filter latency derived from the code
On the board
Camera-to-HDMI latency, output under failing timing, and speed against software on the Zynq’s Arm cores: not measured, needs the board
Reproduce
From the repository root: python3 verify/reports.py; python3 -m unittest discover -s verify; python3 verify/compare.py, which needs Icarus Verilog. The Vivado rebuild steps in the README are untested
Not claimed
A design that meets timing, correct output on the board, a measured end-to-end latency, the textbook Prewitt or Sobel result, a comparison with a software or GPU implementation, or a production-ready design.

What went wrong and limits

The design does not meet timing, and the committed bitstreams were written from the failing implementations. The simulation and a reading of the code found these bugs:

  • the line-buffer write pointer advances every pixel, giving the four-line window above;
  • the horizontal counter’s reset at hsync is overridden by its increment, so it counts from the start of hsync, 192 clocks before active video: at 1080p the rightmost 310 columns (1610–1919) are always white, not the 120 that the buffer width alone suggests, and at 720p every second line is white;
  • gradients are cut to 8 bits, so a strong negative edge is drawn almost black (−253 becomes 3);
  • the magnitude uses wrapped inputs, drives a 17-bit product into a 16-bit signed port, and computes its square root in a latch around a combinational divider, so its cycle behaviour on the board is not defined;
  • syncs and data are delayed by different amounts, shifting the picture 2, 3 or 5 pixels against the syncs, depending on the mode;
  • the buttons are neither synchronised nor debounced.

None is fixed. Each fix needs Vivado re-implementation and a check on the board, and the committed bitstreams, reports and demonstration videos all correspond to the current logic.

The limits: nothing was run in Vivado or on a board. The stimulus is synthetic (noise, squares and single pixels), not natural images, and the simulation assumes an ideal video output core. Whether the camera receiver’s violations also occur in Digilent’s unmodified demo was not checked.

Next

Fix the window, counter, truncation and magnitude bugs, re-implement until timing is met, and check the board’s output against the simulation. Measure camera-to-HDMI latency on the board and compare the filter with a software version on the Zynq’s Arm cores, and only then move this page from draft to published.