Beam学习笔记(3):Flink Streaming Pipeline Translator

前端之家收集整理的这篇文章主要介绍了Beam学习笔记(3):Flink Streaming Pipeline Translator前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。

FlinkStreamingPipelineTranslator

1 所有的可用Translator:

// here you can find all the available translators.
  static {
    TRANSLATORS.put(Read.Bounded.class,new BoundedReadSourceTranslator());
    TRANSLATORS.put(Read.Unbounded.class,new UnboundedReadSourceTranslator());
    TRANSLATORS.put(TextIO.Write.Bound.class,new TextIOWriteBoundStreamingTranslator());

    TRANSLATORS.put(ParDo.MultIoUtput.class,new ParDoStreamingTranslator());
    TRANSLATORS.put(
        SplittableParDo.ProcessElements.class,new SplittableProcessElementsStreamingTranslator());
    TRANSLATORS.put(
        SplittableParDo.GBKIntoKeyedWorkItems.class,new GBKIntoKeyedWorkItemsTranslator());


    TRANSLATORS.put(Window.Assign.class,new WindowAssignTranslator());
    TRANSLATORS.put(Flatten.PCollections.class,new FlattenPCollectionTranslator());
    TRANSLATORS.put(
        FlinkStreamingViewOverrides.CreateFlinkPCollectionView.class,new CreateViewStreamingTranslator());

    TRANSLATORS.put(Reshuffle.class,new ReshuffleTranslatorStreaming());
    TRANSLATORS.put(GroupByKey.class,new GroupByKeyTranslator());
    TRANSLATORS.put(Combine.PerKey.class,new CombinePerKeyTranslator());
  }

2 applyStreamingTransform:

基本逻辑:将PTransform关联到streamingContext

// create the applied PTransform on the streamingContext streamingContext.setCurrentTransform(node.toAppliedPTransform());
    typedTranslator.translateNode(typedTransform,streamingContext);

12种类型的translator多态实现了StreamTransformTranslator.translateNode(T transform,FlinkStreamingTranslationContext context);方法

猜你在找的设计模式相关文章