IMX219 test pattern with window activated randomly

[nzottmann]

In a CM3 / IMX219 based python/picamera (legacy stack) application the camera picture sometimes looks like this.

This happens on a very small number of devices randomly after multiple weeks of continuous runtime with no obvious reason. I have no access to the affected devices, but after research I could reproduce an identical picture by enabling the test pattern of the IMX219 with a window (using manual i2ctransfer commands), which makes me believe the test pattern gets somehow activated in production.

The circular shade probably comes from the ISPs lens correction, so the source must be the sensor itself. When this happens in production, most of the time only a part of the image is pink and the other still shows the camera image. I could reproduce this by modifying the test pattern window registers.

When this problem occurs, it remains until restarting the python application, which restarts the camera module. This resets the registers, which also clears the test mode.

I am currently clueless what could activate the test pattern randomly and rarely. The window is different every time, which looks like random register corruption. But if other registers would be affected, I would expect other problems to appear too, which is not the case. It seems like only some test pattern registers are affected.

I do not have any manual i2c communication to the sensor in my userspace application, only using picamera. As far as I understand, the sensor driver in the legacy stack is part of the firmware, which is not open. Thus my question:

Does the sensor driver in the legacy camera stack contain support for the IMX219 sensor test pattern, which could be triggered by a software bug from userspace?

[6by9]

The firmware never writes register 0x0600 (test_pattern_mode) of the imx219. Certainly nothing ever programs a window for the test pattern.
Which registers are you setting via i2ctransfer?

If I were to hazard a guess I'd suspect I2C corruption, but that is generally very rare. Or possibly power issues.

If you can do i2ctransfer reliably (which isn't guaranteed as you have 2 separate processors trying to access it), then read back the test pattern registers.

The legacy camera stack really is legacy. Support will be minimal.

[nzottmann]

The firmware never writes register 0x0600 (test_pattern_mode) of the imx219. Certainly nothing ever programs a window for the test pattern. Which registers are you setting via i2ctransfer?

I am setting the registers in the range 0x0600 .. 0x0627 to enable solid color test pattern, set the color and the window offset and size, which gave me the image shown above, which looks similar to the ones i got from production devices.

If I were to hazard a guess I'd suspect I2C corruption, but that is generally very rare. Or possibly power issues.

I suspected that too, but what irritates me here is that only the test pattern registers seem to be affected, with random corruption I would expect other errors too.
I cannot rule out power issues of course as this happens only on a very small number of devices (<10 out of >1000) and very rarely (takes multiple weeks), so I cannot reproduce the actual error. In general the power supply should not have any problems as we did lots of verification with multiple worst case load (change) scenarios and have >1000 devices running for >5 years without problems. The issues occurred only recently with devices deployed 1-2 years ago.

If you can do i2ctransfer reliably (which isn't guaranteed as you have 2 separate processors trying to access it), then read back the test pattern registers.

This is currently also the only workaround I can think of. I implemented a retry logic as the reads from userspace sometimes fail on first try, which I expected as the GPU repeatedly writes, probably for AGC. Can you suggest a safe interval for this? every few seconds? Has the GPU always priority over the bus or can parallel access from userspace lead to problems within the GPU - IMX219 communication?

The legacy camera stack really is legacy. Support will be minimal.

I know, this is a legacy product too where we cannot switch to libcamera, as this would require a large rewrite for a discontinued product and we needed some features here the new stack does not provide (90° rotation, more than 2 streams).
We switched to libcamera for the successor, but we still have to provide support for older products and this looks like a bug for the customer. But without knowing the root cause, replacing hardware or parts of the device is not a solution.

The firmware never writes register 0x0600 (test_pattern_mode) of the imx219

This helps me at least to rule out one possible cause. Thank you for your help!

[6by9]

If you want to know the registers used by the firmware driver, https://github.com/raspberrypi/raspiraw/blob/master/imx219_modes.h was based on capturing the I2C commands (at the time we couldn't release them due to NDAs).

Yes the firmware will be updating exposure and gain almost on a per frame basis. There is no way to tell it to back off except to lock AE (there is a MMAL parameter, but I don't recall it immediately).
It'll typically be every 33ms for 30fps, so reading 16 bytes at 100kHz is likely to work (I reckon that is 2ms). You may get away with reading all 40bytes of interest. If memory serves correctly then the ARM generally just times out when a collision does happen.
i2ctransfer -y 10 w2@0x10 0x06 0x00 r16

There is no arbitration at all between the two processors for the I2C peripheral. They will each just start writing registers when they want to, and potentially both respond to interrupts and therefore read the data the other one wanted. This is why it is not supported to have both active at the same time.

[nzottmann]

There is no arbitration at all between the two processors for the I2C peripheral. They will each just start writing registers when they want to, and potentially both respond to interrupts and therefore read the data the other one wanted. This is why it is not supported to have both active at the same time.

This was a good hint for the possible root cause. We have a DS1339 RTC connected to the camera I2C bus. I know this is not ideal, but we had no other choice due to some physical limits and it worked fine during testing. Until now i believed there could not be a conflict as the RTC is handled through a kernel driver.
But if I understand correctly, this can still lead to conflicts as the RTC gets written from the CPU, the camera from the GPU and they write to the same I2C peripheral registers without sync, which would absolutely explain possible corruption.

With this new hint I ran a new test. I started hwclock --systohc in an endless loop (about 20 runs/s). For a static camera picture this runs fine. If the image brightness changes and the camera gain is updated, I see hwclock: RTC_SET_TIME: Connection timed out about once per second, which shows a conflict.
This would explain why the problem appears only rarely in real world, as the RTC is updated only after NTP time sync and a conflict is also rare.

But I would expect to be able to reproduce it faster with running hwclock --systohc in an endless loop. A second script cycled day, hour, minute and second of the date every few seconds to have different write values for the RTC register. I placed a blinking light in front of the camera to keep gain control constantly active. But even after two hours, the camera picture is absolutely fine.

I think the RTC is the way to look at for a possible I2C conflict. Next steps will be looking at hwclock and RTC driver source to understand if the timeout is a missing I2C ack or another error, then perhaps logic analyzer on the bus.

Thank you for your support! This helped me a lot in the right direction. Feel free to close this issue as my initial question is answered this looks like an issue on my side.

[6by9]

Use of the same I2C controller from both the firmware and kernel WILL lead to issues eventually. It's part of the reason why they were referred to as i2c_vc for the VideoCore firmware and i2c_arm for the ARM/Linux controller, denoting ownership. Ignoring that is, unfortunately, courting issues.

The firmware generally has lower interrupt latency than Linux, so if it is expecting an interrupt it can handle the event that the kernel is looking for first, and therefore the kernel times out. The firmware only enables the interrupt handler when it is actively using the peripheral, hence not seeing issues when there are no gain or exposure updates.

With libcamera all the drivers are part of the Linux kernel, and therefore the I2C bus can be shared with other devices (the fan controller and RTC on CM4IO are on the same I2C bus as CAM1 and DISP1)

[nzottmann]

I could finally reproduce the issue. I wrote a script which writes to the RTC registers continuously using ioctl and re-reads them to verify the content. I placed a blinking light in front of the camera to have the AGC update the sensor gain registers as often as possible.

I got about 100 corruptions of the RTC registers (read does not match write, but is a plausible gain value) after about a day of runtime, much more IO and timeout errors for read and write commands and after 1 day of runtime the first image corruption.

It was not the test pattern, 0x0600... range was still fine, but turned out to be the X_ADD_STA_A register, which resulted in a quite similar outcome, with one difference: the unaffected part of the picture still looks normal (which I observed in production), while in windowed test pattern the out-of-window part got dark and more green. Clearing the X_ADD_STA_A register turned the image back to normal.

Thank you again for the help, with your insight it is now clear what happens. With knowing exactly what happens, I will find some kind of workaround.