The four most important steps in the Camera operation are:
- CameraManager -->openCamera --->Open Camera
- CameraDeviceImpl-->createCaptureSession --->Create Capture Session
- CameraCaptureSession -->setRepeatingRequest --->Set Preview Interface
- CameraDeviceImpl-->capture--->Start capturing pictures
Previously, we introduced the openCamera process and the createCaptureSession process as follows:
openCamera Module of Android Camera Principles (1)
openCamera Module of Android Camera Principles (2)
createCaptureSession Module of Android Camera Principles
At this point, the Camera session has been created successfully, and now we can start previewing. After previewing the callback onCaptureCompleted, you can take a picture (callback to onCaptureCompleted, indicating that the full frame data of the capturehas been returned and you can capture the data there).), Because many processes of preview and photo taking are very similar, photo taking is only a node in the preview process, so we will explain preview and photo taking in one article.
1. Preview
The function that initiates the preview is CameraCaptureSession-->setRepeatingRequest. In this article, we will talk about how Camera initiates the preview operation.
CameraCaptureSession-->setRepeatingRequest is a CameraCaptureSession.StateCallback.onConfigured (List<Surface> outputs, CameraCaptureSession.StateCallback callback, Handler handler) that is executed after the output stream configuration is successful in createCaptureSession (List<Surface> outputs) @NonNull Executed in the CameraCaptureSession session function.
mCameraDevice.createCaptureSession(Arrays.asList(surface, mImageReader.getSurface()), new CameraCaptureSession.StateCallback() { @Override public void onConfigured(@NonNull CameraCaptureSession cameraCaptureSession) { // The camera is already closed if (null == mCameraDevice) { return; } // When the session is ready, we start displaying the preview. mCaptureSession = cameraCaptureSession; try { // Auto focus should be continuous for camera preview. mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE); // Flash is automatically enabled when necessary. setAutoFlash(mPreviewRequestBuilder); // Finally, we start displaying the camera preview. mPreviewRequest = mPreviewRequestBuilder.build(); mCaptureSession.setRepeatingRequest(mPreviewRequest, mCaptureCallback, mBackgroundHandler); } catch (CameraAccessException e) { e.printStackTrace(); } } @Override public void onConfigureFailed( @NonNull CameraCaptureSession cameraCaptureSession) { showToast("Failed"); } }, null );
Final Execution
mCaptureSession.setRepeatingRequest(mPreviewRequest, mCaptureCallback, mBackgroundHandler);
Perform the camera preview operation.Operations such as focusing can be done in this onConfigured callback.
- The onConfigured callback indicates that the current configuration stream is complete, the camera is displayed and ready for preview.
- ConfigureFailed configuration failed, Camera Black screen.
public int setRepeatingRequest(CaptureRequest request, CaptureCallback callback, Handler handler) throws CameraAccessException { checkRepeatingRequest(request); synchronized (mDeviceImpl.mInterfaceLock) { checkNotClosed(); handler = checkHandler(handler, callback); return addPendingSequence(mDeviceImpl.setRepeatingRequest(request, createCaptureCallbackProxy(handler, callback), mDeviceExecutor)); } }
- The first parameter, CaptureRequest, identifies the properties of the current capture request, whether to request one or more camera s, whether to reuse previous requests, and so on.
- The second parameter, CaptureCallback, is a catch callback, which is a callback that the developer is directly in contact with.
public interface CaptureCallback { public static final int NO_FRAMES_CAPTURED = -1; public void onCaptureStarted(CameraDevice camera, CaptureRequest request, long timestamp, long frameNumber); public void onCapturePartial(CameraDevice camera, CaptureRequest request, CaptureResult result); public void onCaptureProgressed(CameraDevice camera, CaptureRequest request, CaptureResult partialResult); public void onCaptureCompleted(CameraDevice camera, CaptureRequest request, TotalCaptureResult result); public void onCaptureFailed(CameraDevice camera, CaptureRequest request, CaptureFailure failure); public void onCaptureSequenceCompleted(CameraDevice camera, int sequenceId, long frameNumber); public void onCaptureSequenceAborted(CameraDevice camera, int sequenceId); public void onCaptureBufferLost(CameraDevice camera, CaptureRequest request, Surface target, long frameNumber); }
This needs to be done by the developer himself. How do these callbacks go up? CameraDeviceCallbacks Callback Module of Android Camera Principles , all through the CameraDeviceCallbacks callback.
Let's take a look at how camera calls work
mCaptureSession.setRepeatingRequest
--->CameraDeviceImpl.setRepeatingRequest
--->CameraDeviceImpl.submitCaptureRequest
The third parameter in CameraDeviceImpl.setRepeatingRequest passes in true.This is important because setRepeatingRequest is also executed when CameraDeviceImpl.capture is executed next, where the third parameter is passed in false.The third parameter, boolean repeating, if true, indicates that the current capture is a process, and the camera frame is constantly filling in; if false, it indicates that the current capture is a moment, that is, taking a photo.
public int setRepeatingRequest(CaptureRequest request, CaptureCallback callback, Executor executor) throws CameraAccessException { List<CaptureRequest> requestList = new ArrayList<CaptureRequest>(); requestList.add(request); return submitCaptureRequest(requestList, callback, executor, /*streaming*/true); } private int submitCaptureRequest(List<CaptureRequest> requestList, CaptureCallback callback, Executor executor, boolean repeating) { //...... }
The core work of CameraDeviceImpl.submitCaptureRequest is three steps:
- 1. Verify that the request in the current CaptureRequest list is reasonable: the core is to verify that the Surface bound to the request exists.
- 2. Send request information to the bottom level.
- 3. Bind the request information returned by the underlying layer and the incoming CaptureCallback for subsequent correct callbacks.
Of these three steps, the second is the core work.
1.1 Send captureRequest request to bottom level
SubmitInfo requestInfo; CaptureRequest[] requestArray = requestList.toArray(new CaptureRequest[requestList.size()]); // Convert Surface to streamIdx and surfaceIdx for (CaptureRequest request : requestArray) { request.convertSurfaceToStreamId(mConfiguredOutputs); } requestInfo = mRemoteDevice.submitRequestList(requestArray, repeating); if (DEBUG) { Log.v(TAG, "last frame number " + requestInfo.getLastFrameNumber()); } for (CaptureRequest request : requestArray) { request.recoverStreamIdToSurface(); }
- Execute request.convertSurfaceToStreamId(mConfiguredOutputs); record locally cached surface s and stream s in memory, and binder is transferred to the camera service layer to prevent duplicate requests on the camera service side.
- requestInfo = mRemoteDevice.submitRequestList(requestArray, repeating); this calls directly to the camera service side.Here's what we need to focus on.
- request.recoverStreamIdToSurface(); callback succeeded, clearing data in memory before.
CameraDeviceClient::submitRequest
--->CameraDeviceClient::submitRequestList
This function has a lot of code, many of the previous executions are reuse retrieval before the cache is available, let's focus on the core execution: in the case of preview, the incoming streaming is true, execute above; in the case of photo, execute else below.
err = mDevice->setStreamingRequestList(metadataRequestList, surfaceMapList, &(submitInfo->mLastFrameNumber));
The incoming submitInfo is a callback parameter that returns to the upper level. If it is in a preview state, the current frame data needs to be updated continuously, so each time the latest frame number is updated.
if (streaming) { err = mDevice->setStreamingRequestList(metadataRequestList, surfaceMapList, &(submitInfo->mLastFrameNumber)); if (err != OK) { String8 msg = String8::format( "Camera %s: Got error %s (%d) after trying to set streaming request", mCameraIdStr.string(), strerror(-err), err); ALOGE("%s: %s", __FUNCTION__, msg.string()); res = STATUS_ERROR(CameraService::ERROR_INVALID_OPERATION, msg.string()); } else { Mutex::Autolock idLock(mStreamingRequestIdLock); mStreamingRequestId = submitInfo->mRequestId; } } else { err = mDevice->captureList(metadataRequestList, surfaceMapList, &(submitInfo->mLastFrameNumber)); if (err != OK) { String8 msg = String8::format( "Camera %s: Got error %s (%d) after trying to submit capture request", mCameraIdStr.string(), strerror(-err), err); ALOGE("%s: %s", __FUNCTION__, msg.string()); res = STATUS_ERROR(CameraService::ERROR_INVALID_OPERATION, msg.string()); } ALOGV("%s: requestId = %d ", __FUNCTION__, submitInfo->mRequestId); }
Camera3Device::setStreamingRequestList
--->Camera3Device::submitRequestsHelper
status_t Camera3Device::submitRequestsHelper( const List<const PhysicalCameraSettingsList> &requests, const std::list<const SurfaceMap> &surfaceMaps, bool repeating, /*out*/ int64_t *lastFrameNumber) { ATRACE_CALL(); Mutex::Autolock il(mInterfaceLock); Mutex::Autolock l(mLock); status_t res = checkStatusOkToCaptureLocked(); if (res != OK) { // error logged by previous call return res; } RequestList requestList; res = convertMetadataListToRequestListLocked(requests, surfaceMaps, repeating, /*out*/&requestList); if (res != OK) { // error logged by previous call return res; } if (repeating) { res = mRequestThread->setRepeatingRequests(requestList, lastFrameNumber); } else { res = mRequestThread->queueRequestList(requestList, lastFrameNumber); } //...... return res; }
MRequestThread->setRepeatingRequests (requestList, lastFrameNumber) are executed during preview;
Execute mRequestThread->queueRequestList (requestList, lastFrameNumber) when taking photos;
mRequestThread->setRepeatingRequests
status_t Camera3Device::RequestThread::setRepeatingRequests( const RequestList &requests, /*out*/ int64_t *lastFrameNumber) { ATRACE_CALL(); Mutex::Autolock l(mRequestLock); if (lastFrameNumber != NULL) { *lastFrameNumber = mRepeatingLastFrameNumber; } mRepeatingRequests.clear(); mRepeatingRequests.insert(mRepeatingRequests.begin(), requests.begin(), requests.end()); unpauseForNewRequests(); mRepeatingLastFrameNumber = hardware::camera2::ICameraDeviceUser::NO_IN_FLIGHT_REPEATING_FRAMES; return OK; }
Place the currently submitted CaptureRequest request in the previous preview request queue, inform the HAL layer that there are new request requests, and the HAL layer connection request starts working, continuously outputting information to the upper layer.Here is the RequestThread thread defined in Camera3Device, which ensures that the stream of information is captured continuously during preview and that cameras are in preview.
1.2 Returns request information and CaptureCallback binding
if (callback != null) { mCaptureCallbackMap.put(requestInfo.getRequestId(), new CaptureCallbackHolder( callback, requestList, executor, repeating, mNextSessionId - 1)); } else { if (DEBUG) { Log.d(TAG, "Listen for request " + requestInfo.getRequestId() + " is null"); } }
/** map request IDs to callback/request data */ private final SparseArray<CaptureCallbackHolder> mCaptureCallbackMap = new SparseArray<CaptureCallbackHolder>();
1. Sending a captureRequest request to the bottom level--->The callback's requestIinfo indicates the result of the current capture request and binds the requestInfo.getRequestId() to the CaptureCallbackHolder because the Camera 2 architecture supports sending multiple CaptureRequest requests. Without this binding mechanism, subsequent callbacks can cause serious confusion., or even callbacks can't be made, so developers can't continue to use them.
Let's look at the code where these callbacks are used:
CameraDeviceCallbacks Callback Module of Android Camera Principles It has been explained that CameraDeviceCallbacks.aidl is a callback for the camera service process to communicate with the user process. Inside this callback, take out the CaptureCallback callback bound by CaptureRequest and call into the CaptureCallback callback function so that developers can use it directly.
Here's a look at the onCaptureStarted callback for CameraDeviceCallbacks-->
public void onCaptureStarted(final CaptureResultExtras resultExtras, final long timestamp) { int requestId = resultExtras.getRequestId(); final long frameNumber = resultExtras.getFrameNumber(); if (DEBUG) { Log.d(TAG, "Capture started for id " + requestId + " frame number " + frameNumber); } final CaptureCallbackHolder holder; synchronized(mInterfaceLock) { if (mRemoteDevice == null) return; // Camera already closed // Get the callback for this frame ID, if there is one holder = CameraDeviceImpl.this.mCaptureCallbackMap.get(requestId); if (holder == null) { return; } if (isClosed()) return; // Dispatch capture start notice final long ident = Binder.clearCallingIdentity(); try { holder.getExecutor().execute( new Runnable() { @Override public void run() { if (!CameraDeviceImpl.this.isClosed()) { final int subsequenceId = resultExtras.getSubsequenceId(); final CaptureRequest request = holder.getRequest(subsequenceId); if (holder.hasBatchedOutputs()) { // Send derived onCaptureStarted for requests within the // batch final Range<Integer> fpsRange = request.get(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE); for (int i = 0; i < holder.getRequestCount(); i++) { holder.getCallback().onCaptureStarted( CameraDeviceImpl.this, holder.getRequest(i), timestamp - (subsequenceId - i) * NANO_PER_SECOND/fpsRange.getUpper(), frameNumber - (subsequenceId - i)); } } else { holder.getCallback().onCaptureStarted( CameraDeviceImpl.this, holder.getRequest(resultExtras.getSubsequenceId()), timestamp, frameNumber); } } } }); } finally { Binder.restoreCallingIdentity(ident); } } }
holder = CameraDeviceImpl.this.mCaptureCallbackMap.get(requestId); then call directly
holder.getCallback().onCaptureStarted( CameraDeviceImpl.this, holder.getRequest(i), timestamp - (subsequenceId - i) * NANO_PER_SECOND/fpsRange.getUpper(), frameNumber - (subsequenceId - i));
Simple and clear.
2. Take photos
Developer calls directly if they want to take a photo
mCaptureSession.capture(mPreviewRequestBuilder.build(), mCaptureCallback, mBackgroundHandler);
The call process to take a photo is similar to the preview except that the parameters passed in in the calling function are different.
public int capture(CaptureRequest request, CaptureCallback callback, Executor executor) throws CameraAccessException { if (DEBUG) { Log.d(TAG, "calling capture"); } List<CaptureRequest> requestList = new ArrayList<CaptureRequest>(); requestList.add(request); return submitCaptureRequest(requestList, callback, executor, /*streaming*/false); }
The submitCaptureRequest is also called when taking a photo, but the third parameter passes in false, which means that you don't need to cycle through the frame data on the HAL call to get only the instantaneous frame data.
The difference between a photo and a preview call is:CameraDeviceClient::submitRequestList
if (streaming) { //...... } else { err = mDevice->captureList(metadataRequestList, surfaceMapList, &(submitInfo->mLastFrameNumber)); if (err != OK) { String8 msg = String8::format( "Camera %s: Got error %s (%d) after trying to submit capture request", mCameraIdStr.string(), strerror(-err), err); ALOGE("%s: %s", __FUNCTION__, msg.string()); res = STATUS_ERROR(CameraService::ERROR_INVALID_OPERATION, msg.string()); } ALOGV("%s: requestId = %d ", __FUNCTION__, submitInfo->mRequestId); }
Next call to
mDevice->captureList
--->Camera3Device::submitRequestsHelper
status_t Camera3Device::submitRequestsHelper( const List<const PhysicalCameraSettingsList> &requests, const std::list<const SurfaceMap> &surfaceMaps, bool repeating, /*out*/ int64_t *lastFrameNumber) { //...... RequestList requestList; //...... if (repeating) { res = mRequestThread->setRepeatingRequests(requestList, lastFrameNumber); } else { res = mRequestThread->queueRequestList(requestList, lastFrameNumber); } //...... return res; }
Execute the queueRequestList in the Camera3Device::RequestThread thread.
status_t Camera3Device::RequestThread::queueRequestList( List<sp<CaptureRequest> > &requests, /*out*/ int64_t *lastFrameNumber) { ATRACE_CALL(); Mutex::Autolock l(mRequestLock); for (List<sp<CaptureRequest> >::iterator it = requests.begin(); it != requests.end(); ++it) { mRequestQueue.push_back(*it); } if (lastFrameNumber != NULL) { *lastFrameNumber = mFrameNumber + mRequestQueue.size() - 1; ALOGV("%s: requestId %d, mFrameNumber %" PRId32 ", lastFrameNumber %" PRId64 ".", __FUNCTION__, (*(requests.begin()))->mResultExtras.requestId, mFrameNumber, *lastFrameNumber); } unpauseForNewRequests(); return OK; }
*lastFrameNumber = mFrameNumber + mRequestQueue.size() - 1;
Here is the key execution code to indicate that the latest capture frame data is currently being fetched.
Where do I capture image s when I take pictures?
camera1 provides PictureCallback callbacks to provide real-time preview callbacks, where you can get image data callbacks.
camera2 does not have this interface, but provides ImageReader.OnImageAvailableListener to implement callbacks.
public interface OnImageAvailableListener { /** * Callback that is called when a new image is available from ImageReader. * * @param reader the ImageReader the callback is associated with. * @see ImageReader * @see Image */ void onImageAvailable(ImageReader reader); }
Remember Photo of Android Camera Module Resolution To set before mentioning openCamera
mImageReader = ImageReader.newInstance(largest.getWidth(), largest.getHeight(), ImageFormat.JPEG, /*maxImages*/2); mImageReader.setOnImageAvailableListener( mOnImageAvailableListener, mBackgroundHandler);
ImageReader has a getSurface() function, which is the photo output stream of ImageReader. When we take pictures, we usually have two output streams (the output Surface object), one is the preview stream, and the other is the photo stream.Do not remember to refer to createCaptureSession Module of Android Camera Principles , the photo stream set by ImageReader is set to the camera service side.
public Surface getSurface() { return mSurface; }
Let's see when we can call back this interface.
ImageReader callback interface.jpg
Look at the call process above and call into ImageReader.OnImageAvailableListener->onImageAvailable, we get ImageReader->acquireNextImage to get the collected image.Previews of streaming data are also available in ImageReader.SurfacePlane encapsulates the returned ByteBuffer data for developers to obtain in real time.
private class SurfacePlane extends android.media.Image.Plane { private SurfacePlane(int rowStride, int pixelStride, ByteBuffer buffer) { mRowStride = rowStride; mPixelStride = pixelStride; mBuffer = buffer; /** * Set the byteBuffer order according to host endianness (native * order), otherwise, the byteBuffer order defaults to * ByteOrder.BIG_ENDIAN. */ mBuffer.order(ByteOrder.nativeOrder()); } @Override public ByteBuffer getBuffer() { throwISEIfImageIsInvalid(); return mBuffer; } final private int mPixelStride; final private int mRowStride; private ByteBuffer mBuffer; }
Note: Many developers use Camera.PreviewCallback in camera1
void onPreviewFrame(byte[] data, Camera camera)
Real-time data can be obtained, but there is no such interface in camera2. Although the interface method of camera1 can also be used, the interface replaced by Camera2 is ImageReader.OnImageAvailableListener->onImageAvailable