librestreaming

Introduction: 安卓实时滤镜 RTMP 推流库。支持 cpu 滤镜,gpu 滤镜。音频滤镜。
More: Author   ReportBugs   
Tags:
rtmp-推流-滤镜-android-

This project uses Android lastest MediaCodec API for video/audio encoding and popular C ibrary librtmp (source code included) for rtmp streaming, in addionion, provides ability to implement real-time effect filters after camera capturing phase and before encoding phase. Some features are:

  • Support Android 4.3 and higher (Android 6.0/7.0-preview tested)
  • Filters support soft mode (CPU processing) and hard mode (GPU/OpenGLES rendering)
  • Soft mode filter can be implemented by processing NV21 image array captured from camera
  • Hard mode filter can be implemented by rendering image texture captured from camera
  • Compatible with GPUImage,use GPUImageFilter without change one line
  • Support pixel rotation and flip
  • Support encoding paramaters like resolution, bitrate,fps etc
  • Fast front/rear camera swaping without interrupt rtmp streaming
  • Support draw android View into video

Soft Filter Mode

  • Color format is NV21
  • Create your own filter by extending BaseSoftVideoFilter
  • Filter effect can be manipulated via java, ndk or renderscript
  • Adjust your filter properties between acquireSoftVideoFilter and releaseSoftVideoFilter, call releaseSoftVideoFilter as soon as operation finished (after acquireSoftVideoFilter), interval better be less than 3ms

Hard Filter Mode

  • Create your own filter by extending either BaseHardVideoFilter or OriginalHardVideoFilter
  • OriginalHardVideoFilter is more simple to use, just need to proivde custmom shader to get various effects
  • BaseHardVideoFilter is more powerful and flexible and need write openGLES code by yourself. You can theoretically achieve any effects you want via BaseHardVideoFilter.
  • Adjust your filter properties between acquireHardVideoFilter and releaseHardVideoFilter, call releaseHardVideoFilter as soon as operation finished (after acquireHardVideoFilter), interval better be less than 3ms

Start/stop preview and stream:

    @Override
    public void onCreate(final Bundle savedInstanceState) {
        ... ...
        RESConfig resConfig = RESConfig.obtain();
        resConfig.setRtmpAddr("rtmp://***");
        resConfig.setFilterMode(RESConfig.FilterMode.HARD);
        resConfig.setTargetVideoSize(new Size(720, 480));
        resConfig.setBitRate(1000 * 1024);
        resConfig.setVideoFPS(30);
        resConfig.setVideoGOP(1);
        resConfig.setRenderingMode(RESConfig.RenderingMode.OpenGLES);//setrender mode in softmode
        resConfig.setDefaultCamera(Camera.CameraInfo.CAMERA_FACING_FRONT);
        if (!resClient.prepare(resConfig)) {
            Log.e("Main", "prepare,failed!!");
            finish();
        }
    }

    @Override
    public void onSurfaceTextureAvailable(SurfaceTexture surface, int width, int height) {
        resClient.startPreview(surface, width, height);
    }

    @Override
    public void onSurfaceTextureSizeChanged(SurfaceTexture surface, int width, int height) {
        resClient.updatePreview(width, height);
    }

    @Override
    public boolean onSurfaceTextureDestroyed(SurfaceTexture surface) {
        resClient.stopPreview();
        return false;
    }

    @Override
    protected void onStart() {
        super.onResume();
        resClient.startStreaming();
    }

    @Override
    protected void onStop() {
        super.onPause();
        resClient.stopStreaming();
    }

    @Override
    protected void onDestroy() {
        super.onDestroy();
        resClient.destroy();
    }

Set recording direction:

        int frontDirection, backDirection;
        Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
        Camera.getCameraInfo(Camera.CameraInfo.CAMERA_FACING_FRONT, cameraInfo);
        frontDirection = cameraInfo.orientation;
        Camera.getCameraInfo(Camera.CameraInfo.CAMERA_FACING_BACK, cameraInfo);
        backDirection = cameraInfo.orientation;
        if (this.getResources().getConfiguration().orientation == Configuration.ORIENTATION_PORTRAIT) {
            resConfig.setFrontCameraDirectionMode((frontDirection == 90 ? RESConfig.DirectionMode.FLAG_DIRECTION_ROATATION_270 : RESConfig.DirectionMode.FLAG_DIRECTION_ROATATION_90) | RESConfig.DirectionMode.FLAG_DIRECTION_FLIP_HORIZONTAL);
            resConfig.setBackCameraDirectionMode((backDirection == 90 ? RESConfig.DirectionMode.FLAG_DIRECTION_ROATATION_90 : RESConfig.DirectionMode.FLAG_DIRECTION_ROATATION_270));
        } else {
            resConfig.setBackCameraDirectionMode((backDirection == 90 ? RESConfig.DirectionMode.FLAG_DIRECTION_ROATATION_0 : RESConfig.DirectionMode.FLAG_DIRECTION_ROATATION_180));
            resConfig.setFrontCameraDirectionMode((frontDirection == 90 ? RESConfig.DirectionMode.FLAG_DIRECTION_ROATATION_180 : RESConfig.DirectionMode.FLAG_DIRECTION_ROATATION_0) | RESConfig.DirectionMode.FLAG_DIRECTION_FLIP_HORIZONTAL);
        }

Add filter:

    BlackWhiteSoftFilter bwsFilter = new BlackWhiteSoftFilter();
    resClient.setSoftVideoFilter(bwsFilter);
    BaseSoftVideoFilter filter = resClient.acquireSoftVideoFilter();
    if (filter != null) {
        if (filter instanceof BlackWhiteFilterSoft) {
            BlackWhiteFilterSoft bwsFilter = (BlackWhiteFilterSoft) filter;
            blackWhiteFilter.setGap((byte) 128); // 0 ~ 255
        }
    ... ...
    resClient.releaseSoftVideoFilter();

For more information, please checkout sample code in me.lake.librestreaming.sample.softfilter and me.lake.librestreaming.sample.hardfilter.

简介:

  • 支持 cpu 滤镜和 gpu 滤镜。
  • 硬编码基于 Mediacodec。
  • 推流基于 librtmp。

特性:

  • 支持 cpu 滤镜,并可以通过处理图像数组来自定义滤镜。
  • 支持 gpu 滤镜,并可以通过 opengles 绘制图像纹理来自定义滤镜。
  • gpu 滤镜模式下兼容 GPUImage,一行代码不用修改就可以直接使用 GPUImage 的滤镜。
  • 前后摄像头快速切换,不会打断推流。
  • 可以自定义帧率,最大不会超过设备支持帧率
  • 可以选择图像大小,码流比特率,具体取决于设备支持。

关于滤镜:

支持两种滤镜模式:使用 cpu 的软模式和使用 gpu(opengles)的硬模式,硬模式效率通常高于软模式。

软模式下:

  • 你可以通过继承 BaseSoftVideoFilter 创建自定义滤镜。
  • sample.softfilter 中有一些使用 Java,NDK 和 RenderScript 来处理图像的滤镜示例。
  • 在 acquireSoftVideoFilter 和 releaseSoftVideoFilter 之间可以安全的修改滤镜的属性,不要持有滤镜超过 3 毫秒。
  • 滤镜中的颜色格式为 NV21。

硬模式下:

  • 你可以通过继承 BaseHardVideoFilter 或者 OriginalHardVideoFilter 来创建自定义滤镜。
  • sample.hardfilter 中有一些示例。
  • 继承 OriginalHardVideoFilter 只需要通过提供 Shader 就可以容易的实现大部分效果。
  • 继承 BaseHardVideoFilter 需要自己处理 opengl 细节,但是可以实现任何效果。
  • 在 acquireHardVideoFilter 和 releaseHardVideoFilter 之间可以安全的修改滤镜的属性。不要持有滤镜超过 3 毫秒。

关于磨皮算法:

  • 使用优化过的带阈值的高斯模糊算法(Selective Gaussian Blur),sigma = 0.1,稀疏的取周围 24 个点,正态分布取权重计算均值。

关于兼容 GPUImageFilter:

  • 使 GPUImageCompatibleFilter 来兼容 GPUImage 的滤镜,注意不能直接使用 GPUImageGroupFilter,需要把单独的滤镜用 GPUImageCompatibleFilter 包裹,再用 HardVideoGroupFilter 组合起来,具体可以参考 sample 中的 gpuimage:SketchGroup 滤镜。
Apps
About Me
GitHub: Trinea
Facebook: Dev Tools