The Apache Flink community released the third bugfix version of the Apache Flink 1.11 series.

This release includes 151 fixes and minor improvements for Flink 1.11.2. The list below includes a detailed list of all fixes and improvements.

We highly recommend all users to upgrade to Flink 1.11.3.

Updated Maven dependencies:

<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>1.11.3</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>1.11.3</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>1.11.3</version> </dependency>

You can find the binaries on the updated Downloads page.

List of resolved issues:

Sub-task

• [FLINK-17393] - Improve the FutureCompletingBlockingQueue to wakeup blocking put() more elegantly.
• [FLINK-18604] - HBase ConnectorDescriptor can not work in Table API
• [FLINK-18673] - Calling ROW() in a UDF results in UnsupportedOperationException
• [FLINK-18680] - Improve RecordsWithSplitIds API
• [FLINK-18916] - Add a "Operations" link(linked to dev/table/tableApi.md) under the "Python API" -> "User Guide" -> "Table API" section
• [FLINK-18918] - Add a "Connectors" document under the "Python API" -> "User Guide" -> "Table API" section
• [FLINK-18922] - Add a "Catalogs" link (linked to dev/table/catalogs.md) under the "Python API" -> "User Guide" -> "Table API" section
• [FLINK-18926] - Add a "Environment Variables" document under the "Python API" -> "User Guide" -> "Table API" section
• [FLINK-19162] - Allow Split Reader based sources to reuse record batches
• [FLINK-19205] - SourceReaderContext should give access to Configuration and Hostbame
• [FLINK-20397] - Pass checkpointId to OperatorCoordinator.resetToCheckpoint().

Bug

• [FLINK-9992] - FsStorageLocationReferenceTest#testEncodeAndDecode failed in CI
• [FLINK-13733] - FlinkKafkaInternalProducerITCase.testHappyPath fails on Travis
• [FLINK-15170] - WebFrontendITCase.testCancelYarn fails on travis
• [FLINK-16246] - Exclude "SdkMBeanRegistrySupport" from dynamically loaded AWS connectors
• [FLINK-16268] - Failed to run rank over window with Hive built-in functions
• [FLINK-16768] - HadoopS3RecoverableWriterITCase.testRecoverWithStateWithMultiPart hangs
• [FLINK-17341] - freeSlot in TaskExecutor.closeJobManagerConnection cause ConcurrentModificationException
• [FLINK-17458] - TaskExecutorSubmissionTest#testFailingScheduleOrUpdateConsumers
• [FLINK-17677] - FLINK_LOG_PREFIX recommended in docs is not always available
• [FLINK-17825] - HA end-to-end gets killed due to timeout
• [FLINK-18128] - CoordinatedSourceITCase.testMultipleSources gets stuck
• [FLINK-18196] - flink throws NullPointerException when executeCheckpointing
• [FLINK-18222] - "Avro Confluent Schema Registry nightly end-to-end test" unstable with "Kafka cluster did not start after 120 seconds"
• [FLINK-18815] - AbstractCloseableRegistryTest.testClose unstable
• [FLINK-18818] - HadoopRenameCommitterHDFSTest.testCommitOneFile[Override: false] failed with "java.io.IOException: The stream is closed"
• [FLINK-18836] - Python UDTF doesn't work well when the return type isn't generator
• [FLINK-18915] - FIXED_PATH(dummy Hadoop Path) with WriterImpl may cause ORC writer OOM
• [FLINK-19022] - AkkaRpcActor failed to start but no exception information
• [FLINK-19121] - Avoid accessing HDFS frequently in HiveBulkWriterFactory
• [FLINK-19135] - (Stream)ExecutionEnvironment.execute() should not throw ExecutionException
• [FLINK-19138] - Python UDF supports directly specifying input_types as DataTypes.ROW
• [FLINK-19140] - Join with Table Function (UDTF) SQL example is incorrect
• [FLINK-19151] - Flink does not normalize container resource with correct configurations when Yarn FairScheduler is used
• [FLINK-19154] - Application mode deletes HA data in case of suspended ZooKeeper connection
• [FLINK-19170] - Parameter naming error
• [FLINK-19201] - PyFlink e2e tests is instable and failed with "Connection broken: OSError"
• [FLINK-19227] - The catalog is still created after opening failed in catalog registering
• [FLINK-19237] - LeaderChangeClusterComponentsTest.testReelectionOfJobMaster failed with "NoResourceAvailableException: Could not allocate the required slot within slot request timeout"
• [FLINK-19244] - CSV format can't deserialize null ROW field
• [FLINK-19250] - SplitFetcherManager does not propagate errors correctly
• [FLINK-19253] - SourceReaderTestBase.testAddSplitToExistingFetcher hangs
• [FLINK-19258] - Fix the wrong example of "csv.line-delimiter" in CSV documentation
• [FLINK-19280] - The option "sink.buffer-flush.max-rows" for JDBC can't be disabled by set to zero
• [FLINK-19281] - LIKE cannot recognize full table path
• [FLINK-19291] - Fix exception for AvroSchemaConverter#convertToSchema when RowType contains multiple row fields
• [FLINK-19295] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string
• [FLINK-19300] - Timer loss after restoring from savepoint
• [FLINK-19321] - CollectSinkFunction does not define serialVersionUID
• [FLINK-19338] - New source interface cannot unregister unregistered source
• [FLINK-19361] - Make HiveCatalog thread safe
• [FLINK-19398] - Hive connector fails with IllegalAccessError if submitted as usercode
• [FLINK-19401] - Job stuck in restart loop due to excessive checkpoint recoveries which block the JobMaster
• [FLINK-19423] - Fix ArrayIndexOutOfBoundsException when executing DELETE statement in JDBC upsert sink
• [FLINK-19433] - An Error example of FROM_UNIXTIME function in document
• [FLINK-19448] - CoordinatedSourceITCase.testEnumeratorReaderCommunication hangs
• [FLINK-19535] - SourceCoordinator should avoid fail job multiple times.
• [FLINK-19557] - Issue retrieving leader after zookeeper session reconnect
• [FLINK-19585] - UnalignedCheckpointCompatibilityITCase.test:97->runAndTakeSavepoint: "Not all required tasks are currently running."
• [FLINK-19587] - Error result when casting binary type as varchar
• [FLINK-19618] - Broken link in docs
• [FLINK-19629] - Fix NullPointException when deserializing map field with null value for Avro format
• [FLINK-19675] - The plan of is incorrect when Calc contains WHERE clause, composite fields access and Python UDF at the same time
• [FLINK-19695] - Writing Table with RowTime Column of type TIMESTAMP(3) to Kafka fails with ClassCastException
• [FLINK-19717] - SourceReaderBase.pollNext may return END_OF_INPUT if SplitReader.fetch throws
• [FLINK-19740] - Error in to_pandas for table containing event time: class java.time.LocalDateTime cannot be cast to class java.sql.Timestamp
• [FLINK-19741] - InternalTimeServiceManager fails to restore due to corrupt reads if there are other users of raw keyed state streams
• [FLINK-19748] - KeyGroupRangeOffsets#KeyGroupOffsetsIterator should skip key groups that don't have a defined offset
• [FLINK-19750] - Deserializer is not opened in Kafka consumer when restoring from state
• [FLINK-19755] - Fix CEP documentation error of the example in 'After Match Strategy' section
• [FLINK-19775] - SystemProcessingTimeServiceTest.testImmediateShutdown is instable
• [FLINK-19777] - Fix NullPointException for WindowOperator.close()
• [FLINK-19790] - Writing MAP to Kafka with JSON format produces incorrect data.
• [FLINK-19806] - Job may try to leave SUSPENDED state in ExecutionGraph#failJob()
• [FLINK-19816] - Flink restored from a wrong checkpoint (a very old one and not the last completed one)
• [FLINK-19852] - Managed memory released check can block IterativeTask
• [FLINK-19867] - Validation fails for UDF that accepts var-args
• [FLINK-19894] - Use iloc for positional slicing instead of direct slicing in from_pandas
• [FLINK-19901] - Unable to exclude metrics variables for the last metrics reporter.
• [FLINK-19906] - Incorrect result when compare two binary fields
• [FLINK-19907] - Channel state (upstream) can be restored after emission of new elements (watermarks)
• [FLINK-19909] - Flink application in attach mode could not terminate when the only job is canceled
• [FLINK-19948] - Calling NOW() function throws compile exception
• [FLINK-20013] - BoundedBlockingSubpartition may leak network buffer if task is failed or canceled
• [FLINK-20018] - pipeline.cached-files option cannot escape ':' in path
• [FLINK-20033] - Job fails when stopping JobMaster
• [FLINK-20050] - SourceCoordinatorProviderTest.testCheckpointAndReset failed with NullPointerException
• [FLINK-20063] - File Source requests an additional split on every restore.
• [FLINK-20064] - Broken links in the documentation
• [FLINK-20065] - UnalignedCheckpointCompatibilityITCase.test failed with AskTimeoutException
• [FLINK-20068] - KafkaSubscriberTest.testTopicPatternSubscriber failed with unexpected results
• [FLINK-20069] - docs_404_check doesn't work properly
• [FLINK-20076] - DispatcherTest.testOnRemovedJobGraphDoesNotCleanUpHAFiles does not test the desired functionality
• [FLINK-20079] - Modified UnalignedCheckpointITCase...MassivelyParallel fails
• [FLINK-20081] - ExecutorNotifier should run handler in the main thread when receive an exception from the callable.
• [FLINK-20143] - use yarn.provided.lib.dirs config deploy job failed in yarn per job mode
• [FLINK-20165] - YARNSessionFIFOITCase.checkForProhibitedLogContents: Error occurred during initialization of boot layer java.lang.IllegalStateException: Module system already initialized
• [FLINK-20175] - Avro Confluent Registry SQL format does not support adding nullable columns
• [FLINK-20183] - Fix the default PYTHONPATH is overwritten in client side
• [FLINK-20193] - SourceCoordinator should catch exception thrown from SplitEnumerator.start()
• [FLINK-20194] - KafkaSourceFetcherManager.commitOffsets() should handle the case when there is no split fetcher.
• [FLINK-20200] - SQL Hints are not supported in "Create View" syntax
• [FLINK-20213] - Partition commit is delayed when records keep coming
• [FLINK-20221] - DelimitedInputFormat does not restore compressed filesplits correctly leading to dataloss
• [FLINK-20222] - The CheckpointCoordinator should reset the OperatorCoordinators when fail before the first checkpoint.
• [FLINK-20223] - The RecreateOnResetOperatorCoordinator and SourceCoordinator executor thread should use the user class loader.
• [FLINK-20243] - Remove useless words in documents
• [FLINK-20262] - Building flink-dist docker image does not work without python2
• [FLINK-20266] - New Sources prevent JVM shutdown when running a job
• [FLINK-20270] - Fix the regression of missing ExternallyInducedSource support in FLIP-27 Source.
• [FLINK-20277] - flink-1.11.2 ContinuousFileMonitoringFunction cannot restore from failure
• [FLINK-20284] - Error happens in TaskExecutor when closing JobMaster connection if there was a python UDF
• [FLINK-20285] - LazyFromSourcesSchedulingStrategy is possible to schedule non-CREATED vertices
• [FLINK-20333] - Flink standalone cluster throws metaspace OOM after submitting multiple PyFlink UDF jobs.
• [FLINK-20351] - Execution.transitionState does not properly log slot location
• [FLINK-20382] - Exception thrown from JobMaster.startScheduling() may be ignored.
• [FLINK-20396] - Add "OperatorCoordinator.resetSubtask()" to fix order problems of "subtaskFailed()"
• [FLINK-20404] - ZooKeeper quorum fails to start due to missing log4j library
• [FLINK-20413] - Sources should add splits back in "resetSubtask()", rather than in "subtaskFailed()".
• [FLINK-20418] - NPE in IteratorSourceReader
• [FLINK-20442] - Fix license documentation mistakes in flink-python.jar
• [FLINK-20492] - The SourceOperatorStreamTask should implement cancelTask() and finishTask()
• [FLINK-20554] - The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI

New Feature

• [FLINK-19934] - [FLIP-27 source] add new API: SplitEnumeratorContext.runInCoordinatorThread(Runnable)

Improvement

• [FLINK-16753] - Exception from AsyncCheckpointRunnable should be wrapped in CheckpointException
• [FLINK-18139] - Unaligned checkpoints checks wrong channels for inflight data.
• [FLINK-18500] - Make the legacy planner exception more clear when resolving computed columns types for schema
• [FLINK-18545] - Sql api cannot specify flink job name
• [FLINK-18715] - add cpu usage metric of jobmanager/taskmanager
• [FLINK-19193] - Recommend stop-with-savepoint in upgrade guidelines
• [FLINK-19225] - Improve code and logging in SourceReaderBase
• [FLINK-19245] - Set default queue capacity for FLIP-27 source handover queue to 2
• [FLINK-19251] - Avoid confusing queue handling in "SplitReader.handleSplitsChanges()"
• [FLINK-19252] - Jaas file created under io.tmp.dirs - folder not created if not exists
• [FLINK-19265] - Simplify handling of 'NoMoreSplitsEvent'
• [FLINK-19339] - Support Avro's unions with logical types
• [FLINK-19523] - Hide sensitive command-line configurations
• [FLINK-19569] - Upgrade ICU4J to 67.1+
• [FLINK-19677] - TaskManager takes abnormally long time to register with JobManager on Kubernetes
• [FLINK-19698] - Add close() method and onCheckpointComplete() to the Source.
• [FLINK-19892] - Replace __metaclass__ field with metaclass keyword
• [FLINK-20049] - Simplify handling of "request split".
• [FLINK-20055] - Datadog API Key exposed in Flink JobManager logs
• [FLINK-20142] - Update the document for CREATE TABLE LIKE that source table from different catalog is supported
• [FLINK-20152] - Document which execution.target values are supported
• [FLINK-20156] - JavaDocs of WatermarkStrategy.withTimestampAssigner are wrong wrt Java 8
• [FLINK-20169] - Move emitting MAX_WATERMARK out of SourceOperator processing loop
• [FLINK-20207] - Improve the error message printed when submitting the pyflink jobs via 'flink run'
• [FLINK-20296] - Explanation of keyBy was broken by find/replace of deprecated forms of keyBy

Test

• [FLINK-18725] - "Run Kubernetes test" failed with "30025: provided port is already allocated"

Task

• [FLINK-20455] - Add check to LicenseChecker for top level /LICENSE files in shaded jars