GNSS Receiver Evaluation

Record-and-Playback Test Methods

This article addresses how best to quantify “which navigation system performs best” in a realistic testing scenario. The methodology focuses on land vehicles navigating in urban environments, but applies equally well to pedestrian navigation and can be adapted for testing assisted-GNSS implementations. During a drive test, the truth-reference system and RF recording system log samples to disk, with no need for the receivers under test to be included during the actual drive. 

By Eric Vinande, Brian Weinstein, Tianxing Chu, and Dennis Akos, University of Colorado, Boulder

FIGURE 1. Traditional in-vehicle receiver testing.

Radio frequency record-and-playback systems (RPS) have recently become commercially available. These systems sample the RF environment and store it to disk during a drive test and can replay it through receivers back in the lab environment. Here we explore the improvements in dynamic testing methodology created by these units.

RPS test system installation.

RPS constitute a stark contrast to more traditional signal simulators that use pre-defined trajectories and mathematical models to determine appropriate RF output. Signal simulators attempt to reproduce environmental error factors such as multipath, inertial aiding system errors, and building and vehicle obstructions. They rely on mathematical models to simulate these various error sources. In some cases they do a reasonable job of reproducing these errors, but the dynamic urban environment is so complex (for example, rapidly varying/fading signal strength(s), multiple multipath signals, short/long duration obstructions of multiple layers) that even a sophisticated mathematical model can not replicate all effects completely. Some simulators include software that enables the user to define a trajectory and a limited amount of urban scenario details. Again, only so much realism can be created in a simulation environment. Existing testing standards are simulator-based, and as such, are circumscribed by the signal simulator limitations in representing a dynamic environment.

Positioning performance of a satellite navigation receiver under test (RUT) is coupled with its RF front-end system and local oscillator quality. Because of the variation in RF components between RUTs, some likely have superior RF interference (RFI) immunity. RFI can be a serious issue in certain land vehicles due to on-board electrical systems or because of external interference sources.

This article describes a testing method applicable to all receiver types, and complementary to that described in the December 2009 GPS World article by Mitelman and colleagues, “Testing Software Receivers,” regarding validation testing within a production environment. Added elements include taking into account truth-system uncertainty and a repeatability verification of the RF playback process through non-deterministic hardware receivers.

We present here the dynamic testing approach currently used at the University of Colorado in Boulder for receiver evaluation and comparison in the urban environment. The approach also includes the ability to assess the effect of sensor augmentations (for example, inertial, environmental) on positioning performance.

Truth Reference. Comparison with a truth reference system is essential for evaluation of satellite navigation receivers. For dynamic testing, this typically includes a survey-grade receiver coupled with a tactical-grade (or better) inertial measurement unit (IMU) and associated carrier-phase differential post-processing software. This software is filter-based and provides a positioning-error estimate in various components. Truth reference systems provide a continuous position estimate whose quality can vary depending on factors experienced in the urban environment, including length of full/partial satellite signal outage. In this study, we subtracted the 99th-percentile horizontal positioning error estimate of the truth system from the nominal RUT positioning error at each reporting epoch, as shown in Figure 2.

If the RUT position happens to lie within the truth-system position uncertainty, it is not considered to have any position error.

We focus here on a method to evaluate and compare mass-market, consumer-grade receivers to survey-grade receivers. One difference between these two receiver types is the way they handle the trade-off between accuracy and availability. Consumer receivers strive to provide the user with the highest availability, whereas survey receivers’ goal is to maximize accuracy. As a result, consumer-grade receivers will produce more regular position updates in harsh signal-tracking conditions, but must sacrifice accuracy to do so.

FIGURE 2. RUT position error calculation

Current Testing Standards

The 3GPP defines five independent tests for A-GPS receiver certification. They include tests in the areas of: sensitivity with coarse/fine time assistance, nominal accuracy, dynamic range, multipath performance, and moving scenario/periodic update performance. The last three tests include elements that ostensibly pertain to the urban environment. These tests specify discrete, constant signal power levels for implementation in a hardware signal simulator. The discrepancy between the 3GPP-prescribed signal levels and those observed during actual drive testing is detailed as follows.

The 3GPP moving scenario/periodic update performance test trajectory is shown in Figure 3.

FIGURE 3. 3GPP dynamic testing
trajectory (van Diggelen, A-GPS: Assisted
GPS, GNSS, and SBAS, Artech House)

This test profile calls for the simulation of five satellites with a constant signal strength of 2130 dBm while the vehicle travels around the racetrack trajectory. In contrast, during an actual drive test in an urban area, a receiver reported the distribution of carrier-to-noise-density values for all tracked satellites as shown in Figure 4. This more accurately shows the range of signal strengths that should be expected in urban conditions.

FIGURE 4. Drive-test C/N₀ distribution

The 3GPP moving test is considered passed if positions are reported regularly, and 95 percent of them are within 100 meters of the true position. This is not a particularly difficult test for a RUT to retain signal lock through, as the linear acceleration is about 0.15 g and the centripetal acceleration is about 0.25 g.

It is difficult for independent third parties to carry out a receiver evaluation following 3GPP guidelines as several of the tests require receiver restarts, which in turn requires testing automation. Depending on the receiver-evaluation hardware availability, restart commands may not be available to to an independent evaluator.

3GPP receiver testing results are quoted as pass or fail over a large number of short evaluations. For the dynamic environment, the system performance over continuous time is required to make a proper comparison between evaluated receivers.

In general, evaluating the GPS engines embedded within cell phones or other devices is difficult. Most are not made to interface with an external antenna, and the mere act of adding an antenna connection can significantly alter performance. The output format is not always documented, if it is even available to an end user. To allow fair across-the-board comparisons, GPS chipset manufacturers should make available development kits that have external antenna connections and well-documented message output formats.

Drive-Test Configuration

Visualization Methods

In addition to quantitative methods, we have created a qualitative visualization to assist with interpretation of the raw data. The same parsed data sets that provide the statistical script input are fed into a viewer script along with the post-processed truth reference data. With the truth-reference system data plotted in the center of the screen, each RUT is then plotted the correct distance and direction away, based on the distance and direction of error compared to truth. The receiver plots are overlaid onto Google Earth images centered on the truth-reference location. Plots of number of satellites utilized (top right of Figure 5) and elevation (middle right) as reported by each receiver and the sampled RF spectrum (lower right) are also included.

For each reporting epoch, based on the data frequency of the truth-reference system, a frame is generated with the aforementioned characteristics. These frames are gathered and encoded into a movie clip which can then be used as a quick and simple qualitative tool for receiver comparison. Figure 5 shows an individual movie frame. A forward-looking camera capability is also being added to this movie so the test environment can be documented from multiple angles.

FIGURE 5. Movie visualization screenshot

While observing this movie, variations in the sampled RF spectrum from interference or blockages can be associated with the current landscape. Locations of RFI sources can be identified and avoided (or included) in future testing. These RFI and significant blockage locations are of interest for receiver RF component and navigation filter development. The next three figures show spectrum snapshots during various parts of a drive test. In Figure 6, the cumulative GPS spectra rises above the noise floor and is visible during open sky conditions. While below ground level, Figure 7 shows only the front-end filter shape (and relatively minor RFI). Figure 8 shows an example of severe RFI when near a specific parking garage location.

FIGURE 6. Open-sky spectrum (centered
on 1575.42 MHz)

FIGURE 7. Spectrum while below ground
level (centered on 1575.42 MHz).

FIGURE 8. Spectrum near interference
source (centered on 1575.42 MHz).

Record/Playback Concept

To overcome the limitations of hardware signal simulators and repeated vehicle drive testing, the RF record/playback testing method is utilized at the university. Commercially available equipment, capable of recording and playing back an RF signal, has recently become available. Equipment options exist for between $10,000–100,000, with 1–16 bit sampling and 4–25 MHz front-end bandwidth.

Figures 9 and 10 show the concept of “record once, playback many times.” During a drive test, the truth-reference system and RF recording system log samples to disk. There is no need for the RUT to be included during the actual drive test.

FIGURE 9. Recording mode block diagram.

FIGURE 10. Playback
mode block diagram

In the laboratory, the logged RF samples are replayed through a splitter to all RUT. The effect of receiver configuration changes can be evaluated without having to repeat the drive test. At a later time, additional receivers can also be tested using the same stored RF sample file.

During separate record and playback phases, testing considerations and methods discussed previously are implemented.

Since the recording process can only obviously capture current conditions, additional drive-test collections are required if different satellite geometry is desired, or if additional representative antennas need to be evaluated.

Repeatability of RPS Testing

To validate that the playback signal levels were not significantly different from live signals, we conducted an urban, dynamic evaluation. Figure 11 shows that there is typically not more than a 1 dB difference in reported C/N₀ between live and playback modes when testing a receiver that only reported integer values. The two dropout instances were excursions into parking garages.

FIGURE 11. Live and playback C/N₀ values

Figure 12 compares the navigation statistics between replays, using the same five playbacks as in Figure 11. The playbacks show a 1-sigma horizontal position solution spread under 1 meter for approximately 83 percent of the test.

FIGURE 12. Playback Horizontal Position Error Spread.

These two figures verify the repeatability of the RPS testing method and solidify it as an alternative to both signal-simulator testing and live testing of satellite navigation receivers.

Denver Testing Method

To evaluate the RPS concept, we conducted tests in three locations: Boulder, Denver, and Interstate Highway 70, all in Colorado. The Boulder and Denver locations were urban collections, while the Interstate 70 location was a natural canyon with significant elevation change. The collection at each location was repeated with two different representative antennas (patch and cell phone) at nearly the same sidereal time in order to keep the overhead satellite constellation similar.

We examine here the November 11 and 16 Denver tests. The November 11 test used a patch antenna that places nearly all its gain in the upward direction, making it more immune to interfering sources below and to its sides. Figure 13 shows the patch antenn
a location on the van, as well as the truth-system antenna location utilized for testing on both days.

FIGURE 13. Patch antenna (dashboard) and
truth-system antenna (rooftop) locations.

The November 16 test used a cell-phone GPS antenna that does not have a preferential gain direction, making it more susceptible to interfering sources below and to its sides. This antenna type is representative of the typical low-cost antenna (in some cases as simple as a piece of wire) found in consumer cell phones. Figure 14 shows the cell-phone antenna suction-cup mounted to the front window of the testing van. The representative antenna mounting location was chosen to minimize locally-generated RFI effects while also being representative of a typical vehicle-use case.

FIGURE 14. Cell-phone antenna location.

The required equipment and connections are minimal when performing RPS drive testing, as no RUTs are included. The inset to Figure 1 at the beginning of this article shows the RPS unit in the rear of the van, mounted on layers of foam to reduce vibration, which, if not properly addressed, can cause errors in mechanical hard drives writing data at high rates. Also visible are the truth receiver on the center of the van floor, and the car batteries for powering it and the IMU. The IMU is mounted to the vehicle frame and is not shown.

The test drive trajectory through Denver on November 11 and 16 as reported by the truth system is shown in black in Figure 15 and is also repeated in Figures 16 and 17. The test lasted approximately 40 minutes on both days. It started in the upper left part of Figure 15 and continued zig-zagging through downtown to the lower right.

FIGURE 15. Truth trajectory for November 11 and 16 tests.

Figures 16 and 17 show particularly difficult blocks for the four receivers tested under the replay method. These receivers are denoted A (green), B (blue), C (red), and D (yellow).

FIGURE 16. Difficult block #1 during November 11 test and truth
system antenna (rooftop) locations.

The horizontal positioning error statistics for two receivers on the November 11 test are shown in Figures 18 and 19. The left side shows horizontal error in two different zoom levels. The right side shows a histogram and cumulative distribution of errors, and several reporting metrics over the entire test. Even though receiver A in general outperformed receiver B, from the error time histories there are noticeable periods where both receivers simultaneously had positioning difficulties.

FIGURE 17. Difficult block #2 during November 11 test.

Table 1 summarizes the horizontal positioning statistics for all receivers during both tests. Positioning accuracy was severely degraded when replaying samples collected with the cell-phone antenna as compared to the patch antenna. Receiver A was the most accurate across both tests, while receiver B was the least accurate. The uncertainty of the truth system was subtracted out when producing the horizontal positioning results for all receivers.

Table 1

Conclusions

The record-and-playback system testing approach, in our opinion, represents the best way to test hardware receivers. It overcomes the fidelity limits of simulator-based testing, especially when considering the difficult-to-model urban environment. During receiver development, it requires only a single drive test for each location, as sampled RF data can be replayed from disk.

FIGURE 18. Receiver A horizontal positioning error statistics (November 11 test).

FIGURE 19. Receiver B horizontal positioning error statistics (November 11 test).

Having demonstrated that RPS testing is repeatable, we have produced a library of RF sample files representing real-world conditions for continued receiver development and testing purposes.

• Eric Vinande is Ph.D. student at the University of Colorado studying GPS/MEMS inertial sensor integration and urban RFI aspects.
• Brian Weinstein is a BSEE student participating in the Undergraduate Research Opportunity Program for GNSS receiver testing at the University of Colorado.
• Tianxing Chu is a visiting researcher at the University of Colorado from Peking University where he is a Ph.D. student.
• Dennis Akos is an associate professor within the Aerospace Engineering Sciences Department at the University of Colorado with concurrent appointments at Stanford University and Luleå University of Technology.

Manufacturers

Development of the methodology described here used two different RPS systems, one from LabSat (RaceLogic) and one from Averna. The test data come from the Averna system.