Given the code fragment:

import pyspark.pandas as ps
psdf = ps.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
Which method is used to convert a Pandas API on Spark DataFrame (pyspark.pandas.DataFrame) into a standard PySpark DataFrame (pyspark.sql.DataFrame)?

• A. psdf.to_pandas()
• B. psdf.to_pyspark()
• C. psdf.to_spark()
• D. psdf.to_dataframe()

43 of 55.
An organization has been running a Spark application in production and is considering disabling the Spark History Server to reduce resource usage.
What will be the impact of disabling the Spark History Server in production?

• A. Enhanced executor performance due to reduced log size
• B. Improved job execution speed due to reduced logging overhead
• C. Prevention of driver log accumulation during long-running jobs
• D. Loss of access to past job logs and reduced debugging capability for completed jobs

7 of 55.
A developer has been asked to debug an issue with a Spark application. The developer identified that the data being loaded from a CSV file is being read incorrectly into a DataFrame.
The CSV file has been read using the following Spark SQL statement:
CREATE TABLE locations
USING csv
OPTIONS (path '/data/locations.csv')
The first lines of the command SELECT * FROM locations look like this:
| city | lat | long |
| ALTI Sydney | -33... | ... |
Which parameter can the developer add to the OPTIONS clause in the CREATE TABLE statement to read the CSV data correctly again?

• A. 'header' 'false'
• B. 'sep' '|'
• C. 'sep' ','
• D. 'header' 'true'

A data engineer is building an Apache Spark™ Structured Streaming application to process a stream of JSON events in real time. The engineer wants the application to be fault-tolerant and resume processing from the last successfully processed record in case of a failure. To achieve this, the data engineer decides to implement checkpoints.
Which code snippet should the data engineer use?

• A. query = streaming_df.writeStream \
  .format("console") \
  .outputMode("append") \
  .start()
• B. query = streaming_df.writeStream \
  .format("console") \
  .outputMode("complete") \
  .start()
• C. query = streaming_df.writeStream \
  .format("console") \
  .option("checkpoint", "/path/to/checkpoint") \
  .outputMode("append") \
  .start()
• D. query = streaming_df.writeStream \
  .format("console") \
  .outputMode("append") \
  .option("checkpointLocation", "/path/to/checkpoint") \
  .start()