Stedos/nise
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merged mari/konata into nise-backend, adding basic replay compression

Browse Source
This commit is contained in:
nise.moe 2024-06-10 20:35:39 +02:00
parent f84c955523
commit 5fbdfaa322
31 changed files with 305 additions and 482 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

108
konata/pom.xml
View File

@ -1,108 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.nisemoe</groupId>
<artifactId>konata</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<java.version>21</java.version>
<kotlin.version>1.9.22</kotlin.version>
</properties>
<build>
<sourceDirectory>src/main/kotlin</sourceDirectory>
<testSourceDirectory>src/test/kotlin</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<version>${kotlin.version}</version>
<executions>
<execution>
<id>compile</id>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>test-compile</id>
<phase>test-compile</phase>
<goals>
<goal>test-compile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>
<configuration>
<mainClass>MainKt</mainClass>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<!-- For LZMA decompression of replay data -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
<version>1.25.0</version>
</dependency>
<dependency>
<groupId>org.tukaani</groupId>
<artifactId>xz</artifactId>
<version>1.9</version>
</dependency>
<!-- Vector computations -->
<dependency>
<groupId>org.jetbrains.bio</groupId>
<artifactId>viktor</artifactId>
<version>1.2.0</version>
</dependency>
<!-- Math computations -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
<version>3.6.1</version>
</dependency>
<!-- Coroutines -->
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-core</artifactId>
<version>1.7.3</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-test-junit5</artifactId>
<version>${kotlin.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>5.10.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib</artifactId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</project>

111
konata/readme.md
View File

@ -1,111 +0,0 @@
# konata
>osu! utility lib in kotlin for fast replay comparison with multithreading support
This module has the specific purpose of **high-throughput** replay comparison, and only works with replay data as supplied by the osu!api; it does not work with .osr files.
[circleguard](https://github.com/circleguard/circleguard) is a better tool if you are looking for a more complete solution, as it has a GUI and supports .osr files.
this module was built with a narrow task in mind, and I do not have plans to implement more features (especially if circleguard already covers them)
# Usage
### Replay data class
`Replay` is the main data class you'll be throwing around. The only required field is the replay data (verbatim as fetched by the osu!api) in string format.
You can also pass additional parameters:
| parameter | type | required? | notes |
|-----------|------|------------------------------|-------------------------------------------------------------------------------------------------------------|
| id | Long | not for pairs, yes for sets* | used to find the replay in the output, does NOT have to match osu!api, it can be any identifier you'd like. |
| mods | Int | no (defaults to NoMod) | exact value as fetched by the osu!api, it's used to flip the replay y-axis when HR is enabled. |
*You are forced to set the id when using the replay in a set comparison, as it is the identifier that will allow you to match the input to the results.
Example:
```kotlin
// Simplest replay
val replay: Replay = Replay(replayString)
// A NoMod replay with id 1
val replay: Replay = Replay(replayString, id = 1, mods = 0)
// A HDHR (24) replay with id 2
val replay: Replay = Replay(replayString, id = 2, mods = 24)
```
### Replay pairs (2 replays)
The replay strings must be exactly as provided by the osu!api replay endpoint.
The following code calculates the similarity ratio and correlation ratio between two replays, without specifying any mods.
```kotlin
// Compare using objects
val replay1: Replay = Replay(replay1String)
val replay2: Replay = Replay(replay2String)
val result: ReplayPairComparison = compareReplayPair(replay1, replay2)
println(result.similarity) // 20.365197244184895
println(result.correlation) // 0.9770151700235653
// You can also pass the replay data directly as strings
val similarity: ReplayPairComparison = compareReplayPair(replay1String, replay2String)
println(result.similarity) // 20.365197244184895
println(result.correlation) // 0.9770151700235653
```
### Replay sets (n replays)
If we decide to pass a list of replays, there will be optimizations such as multi-threading involved, which can speed up the calculations.
When comparing sets, you *must* set the replay id (it does not have to match the osu! replay id), as it is the identifier that will
allow you to match the input to the results.
```kotlin
// Compare using objects
val replays: Array<Replay> = arrayOf(
Replay("...", id = 1),
Replay("...", id = 2)
)
val result: List<ReplaySetComparison> = compareReplaySet(replays)
println(result[0].replay1Id) // 1
println(result[0].replay2Id) // 2
println(result[0].similarity) // 155.20954003316618
println(result[0].correlation) // 0.9859198745055805
```
By default, the `compareReplaySet` method will default to using as many threads as there are cores on your system.
You can change this behaviour by manually passing an amount of cores to use:
```kotlin
compareReplaySet(replays, numThreads=4)
```
# Benchmarks
### Performance
On my development machine (5900X), the following benchmarks were obtained.
I processed 10 batches of 100 replays each. The min/max/avg time refer to single batches.
| | version | min | max | avg | total | pairs/second |
|-------------|-------------|------|------|------|-------|--------------|
| | v20240211 | 3.1s | 4.2s | 3.3s | 32.7s | 1501/s |
| | v20240211v2 | 2.5s | 3.7s | 2.7s | 26.7s | 1843/s |
| **current** | v20240211v3 | 1.1s | 2.1s | 1.3s | 13.0s | 3789/s |
### Accuracy (compared to Circleguard)
>as of the last version, konata and circleguard give the same results, with a neglibile margin of error.
After selecting a random dataset of ~50,000 osu!std replays for different beatmaps, I compared the results from konata to circleguard, using the latter as the ground truth.
| metric | avg. delta | std. dev. | median | min | max |
|---------------|------------|------------|-----------|-----------|-----------|
| `SIMILARITY` | 0 | 0.000033 | 0 | -0.005373 | 0.007381 |
| `CORRELATION` | -0.000643 | 0.001342 | -0.000433 | -0.041833 | 0.026300 |

1
mari/.github/FUNDING.yml vendored
View File

@ -1 +0,0 @@
patreon: nise_moe

112
mari/pom.xml
View File

@ -1,112 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.nisemoe</groupId>
<artifactId>mari</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<java.version>21</java.version>
<kotlin.version>1.9.22</kotlin.version>
</properties>
<build>
<sourceDirectory>src/main/kotlin</sourceDirectory>
<testSourceDirectory>src/test/kotlin</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<version>${kotlin.version}</version>
<executions>
<execution>
<id>compile</id>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>test-compile</id>
<phase>test-compile</phase>
<goals>
<goal>test-compile</goal>
</goals>
</execution>
</executions>
<configuration>
<compilerPlugins>
<plugin>kotlinx-serialization</plugin>
</compilerPlugins>
</configuration>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-serialization</artifactId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>
<configuration>
<mainClass>MainKt</mainClass>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<!-- For Brotli compression of Judgement data -->
<dependency>
<groupId>com.aayushatharva.brotli4j</groupId>
<artifactId>brotli4j</artifactId>
<version>1.16.0</version>
</dependency>
<!-- For LZMA decompression of replay data -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
<version>1.25.0</version>
</dependency>
<dependency>
<groupId>org.tukaani</groupId>
<artifactId>xz</artifactId>
<version>1.9</version>
</dependency>
<!-- For JSON serialization -->
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-serialization-json</artifactId>
<version>1.6.3</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-test-junit5</artifactId>
<version>${kotlin.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>5.10.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib</artifactId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</project>

29
mari/readme.md
View File

@ -1,29 +0,0 @@
# mari
>osu! utility lib in kotlin to manipulate replays and judgement data in a safe and performant way.
This module allows [nise.moe](https://nise.moe) to juggle a ton of replays and data around.
# Usage
### Compress / decompress judgement data
The `Judgement` data class ought to represent the way a player has played a specific beatmap. It contains the hit error, distance, etc for each hit object. The structure is based off the Circleguard `Investigations.judgements` return type.
You can use `CompressJudgements.compress` and `CompressJudgements.decompress` to losslessly store and retrieve judgement data. According to my estimates, the compressed data is about 33% the size of the original data.
### Decode a replay
>This method is fundamentally written with the assumption that it'll be used on user-provided replays. It is thus designed to be safe and to not crash on invalid replays.
`OsuReplay` allows you to safely decode an `.osr` file. Once you've parsed the file, you can instantiate a new class:
```kotlin
val replay = OsuReplay(replayFile.bytes)
```
If everything goes well, you can then start reading the replay data.
```kotlin
println(replay.playerName) // mrekk
```

5
nise-backend/Build.sh
View File

@ -17,11 +17,6 @@ IMAGE_VERSION="latest"
# Clean up previous build artifacts # Clean up previous build artifacts
rm -rf target/ rm -rf target/
# Build subdependencies
echo "Building subdependencies..."
(cd ../mari && mvn clean install) || { echo "Building mari failed"; exit 1; }
(cd ../konata && mvn clean install) || { echo "Building konata failed"; exit 1; }
# Clean and build the Maven project # Clean and build the Maven project
echo "Building main project..." echo "Building main project..."
mvn clean package || { echo "Maven build failed"; exit 1; } mvn clean package || { echo "Maven build failed"; exit 1; }

50
nise-backend/pom.xml
View File

@ -58,17 +58,6 @@
<version>${testcontainers.version}</version> <version>${testcontainers.version}</version>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency>
<groupId>org.nisemoe</groupId>
<artifactId>konata</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.nisemoe</groupId>
<artifactId>mari</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<dependency> <dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId> <groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-xml</artifactId> <artifactId>jackson-dataformat-xml</artifactId>
@ -140,6 +129,45 @@
<artifactId>kotlin-stdlib</artifactId> <artifactId>kotlin-stdlib</artifactId>
</dependency> </dependency>
<!-- Vector computations -->
<dependency>
<groupId>org.jetbrains.bio</groupId>
<artifactId>viktor</artifactId>
<version>1.2.0</version>
</dependency>
<!-- Math computations -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
<version>3.6.1</version>
</dependency>
<!-- For LZMA decompression of replay data -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
<version>1.26.1</version>
</dependency>
<dependency>
<groupId>org.tukaani</groupId>
<artifactId>xz</artifactId>
<version>1.9</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-test-junit5</artifactId>
<version>${kotlin.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>5.10.0</version>
<scope>test</scope>
</dependency>
<dependency> <dependency>
<groupId>org.springframework.boot</groupId> <groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId> <artifactId>spring-boot-starter-test</artifactId>

4
nise-backend/src/main/kotlin/com/nisemoe/nise/controller/UploadReplayController.kt
View File

@ -5,10 +5,10 @@ import com.nisemoe.generated.tables.references.BEATMAPS
import com.nisemoe.generated.tables.references.SCORES import com.nisemoe.generated.tables.references.SCORES
import com.nisemoe.generated.tables.references.USER_SCORES import com.nisemoe.generated.tables.references.USER_SCORES
import com.nisemoe.generated.tables.references.USER_SCORES_SIMILARITY import com.nisemoe.generated.tables.references.USER_SCORES_SIMILARITY
import com.nisemoe.konata.Replay
import com.nisemoe.konata.compareSingleReplayWithSet
import com.nisemoe.nise.database.BeatmapService import com.nisemoe.nise.database.BeatmapService
import com.nisemoe.nise.integrations.CircleguardService import com.nisemoe.nise.integrations.CircleguardService
import com.nisemoe.nise.konata.Replay
import com.nisemoe.nise.konata.compareSingleReplayWithSet
import com.nisemoe.nise.osu.OsuApi import com.nisemoe.nise.osu.OsuApi
import com.nisemoe.nise.scheduler.ImportScores import com.nisemoe.nise.scheduler.ImportScores
import org.jooq.DSLContext import org.jooq.DSLContext

13
nise-backend/src/main/kotlin/com/nisemoe/nise/database/ScoreService.kt
View File

@ -5,11 +5,10 @@ import com.nisemoe.generated.tables.records.ScoresJudgementsRecord
import com.nisemoe.generated.tables.records.ScoresRecord import com.nisemoe.generated.tables.records.ScoresRecord
import com.nisemoe.generated.tables.references.* import com.nisemoe.generated.tables.references.*
import com.nisemoe.nise.* import com.nisemoe.nise.*
import com.nisemoe.nise.integrations.CircleguardService
import com.nisemoe.nise.osu.Mod import com.nisemoe.nise.osu.Mod
import com.nisemoe.nise.osu.OsuApi import com.nisemoe.nise.osu.OsuApi
import com.nisemoe.nise.service.AuthService import com.nisemoe.nise.service.AuthService
import org.apache.commons.compress.compressors.lzma.LZMACompressorInputStream import com.nisemoe.nise.service.CompressReplay
import org.jooq.Condition import org.jooq.Condition
import org.jooq.DSLContext import org.jooq.DSLContext
import org.jooq.Record import org.jooq.Record
@ -20,7 +19,6 @@ import org.nisemoe.mari.judgements.Judgement
import org.slf4j.LoggerFactory import org.slf4j.LoggerFactory
import org.springframework.stereotype.Service import org.springframework.stereotype.Service
import java.time.LocalDateTime import java.time.LocalDateTime
import java.util.*
import kotlin.math.roundToInt import kotlin.math.roundToInt
@Service @Service
@ -82,9 +80,9 @@ class ScoreService(
.where(SCORES.REPLAY_ID.eq(replayId)) .where(SCORES.REPLAY_ID.eq(replayId))
.fetchOne() ?: return null .fetchOne() ?: return null
val replayData = result.get(SCORES.REPLAY, String::class.java) ?: return null val replayData = result.get(SCORES.REPLAY, ByteArray::class.java) ?: return null
val replay = decompressData(replayData) val replay = CompressReplay.decompressReplay(replayData)
var beatmapFile = result.get(BEATMAPS.BEATMAP_FILE, String::class.java) var beatmapFile = result.get(BEATMAPS.BEATMAP_FILE, String::class.java)
if(beatmapFile == null) { if(beatmapFile == null) {
@ -111,11 +109,6 @@ class ScoreService(
) )
} }
private fun decompressData(replayString: String): ByteArray =
Base64.getDecoder().decode(replayString).inputStream().use { byteStream ->
LZMACompressorInputStream(byteStream).readBytes()
}
fun getReplayData(replayId: Long): ReplayData? { fun getReplayData(replayId: Long): ReplayData? {
val result = dslContext.select( val result = dslContext.select(
SCORES.ID, SCORES.ID,

0
mari/src/main/kotlin/org/nisemoe/mari/judgements/CompressJudgements.kt → nise-backend/src/main/kotlin/com/nisemoe/nise/judgements/CompressJudgements.kt
View File

0
mari/src/main/kotlin/org/nisemoe/mari/judgements/JudgementModel.kt → nise-backend/src/main/kotlin/com/nisemoe/nise/judgements/JudgementModel.kt
View File

6
konata/src/main/kotlin/com/nisemoe/konata/CompareReplayPair.kt → nise-backend/src/main/kotlin/com/nisemoe/nise/konata/CompareReplayPair.kt
View File

@ -1,7 +1,7 @@
package com.nisemoe.konata package com.nisemoe.nise.konata
import com.nisemoe.konata.algorithms.calculateCorrelation import com.nisemoe.nise.konata.algorithms.calculateCorrelation
import com.nisemoe.konata.algorithms.calculateDistance import com.nisemoe.nise.konata.algorithms.calculateDistance
import org.jetbrains.bio.viktor.F64Array import org.jetbrains.bio.viktor.F64Array

2
konata/src/main/kotlin/com/nisemoe/konata/CompareReplaySet.kt → nise-backend/src/main/kotlin/com/nisemoe/nise/konata/CompareReplaySet.kt
View File

@ -1,4 +1,4 @@
package com.nisemoe.konata package com.nisemoe.nise.konata
import kotlinx.coroutines.asCoroutineDispatcher import kotlinx.coroutines.asCoroutineDispatcher
import kotlinx.coroutines.coroutineScope import kotlinx.coroutines.coroutineScope

6
konata/src/main/kotlin/com/nisemoe/konata/Replay.kt → nise-backend/src/main/kotlin/com/nisemoe/nise/konata/Replay.kt
View File

@ -1,7 +1,7 @@
package com.nisemoe.konata package com.nisemoe.nise.konata
import com.nisemoe.konata.tools.getEvents import com.nisemoe.nise.konata.tools.getEvents
import com.nisemoe.konata.tools.processReplayData import com.nisemoe.nise.konata.tools.processReplayData
import org.jetbrains.bio.viktor.F64Array import org.jetbrains.bio.viktor.F64Array
class Replay(string: String, id: Long? = null, mods: Int = 0) { class Replay(string: String, id: Long? = null, mods: Int = 0) {

2
konata/src/main/kotlin/com/nisemoe/konata/ReplayDto.kt → nise-backend/src/main/kotlin/com/nisemoe/nise/konata/ReplayDto.kt
View File

@ -1,4 +1,4 @@
package com.nisemoe.konata package com.nisemoe.nise.konata
data class ReplayPairComparison( data class ReplayPairComparison(
val similarity: Double, val similarity: Double,

2
konata/src/main/kotlin/com/nisemoe/konata/algorithms/Correlation.kt → nise-backend/src/main/kotlin/com/nisemoe/nise/konata/algorithms/Correlation.kt
View File

@ -1,4 +1,4 @@
package com.nisemoe.konata.algorithms package com.nisemoe.nise.konata.algorithms
import kotlinx.coroutines.* import kotlinx.coroutines.*
import org.apache.commons.math3.stat.descriptive.rank.Median import org.apache.commons.math3.stat.descriptive.rank.Median

2
konata/src/main/kotlin/com/nisemoe/konata/algorithms/Distance.kt → nise-backend/src/main/kotlin/com/nisemoe/nise/konata/algorithms/Distance.kt
View File

@ -1,4 +1,4 @@
package com.nisemoe.konata.algorithms package com.nisemoe.nise.konata.algorithms
import org.jetbrains.bio.viktor.F64Array import org.jetbrains.bio.viktor.F64Array
import org.jetbrains.bio.viktor._I import org.jetbrains.bio.viktor._I

6
konata/src/main/kotlin/com/nisemoe/konata/tools/DecodeReplay.kt → nise-backend/src/main/kotlin/com/nisemoe/nise/konata/tools/DecodeReplay.kt
View File

@ -1,6 +1,6 @@
package com.nisemoe.konata.tools package com.nisemoe.nise.konata.tools
import com.nisemoe.konata.ReplayEvent import com.nisemoe.nise.konata.ReplayEvent
import org.apache.commons.compress.compressors.lzma.LZMACompressorInputStream import org.apache.commons.compress.compressors.lzma.LZMACompressorInputStream
import java.util.* import java.util.*
import kotlin.collections.ArrayList import kotlin.collections.ArrayList
@ -16,7 +16,7 @@ private fun decompressData(replayString: String): ByteArray =
LZMACompressorInputStream(byteStream).readBytes() LZMACompressorInputStream(byteStream).readBytes()
} }
internal fun processEvents(replayDataStr: String): ArrayList<ReplayEvent> { fun processEvents(replayDataStr: String): ArrayList<ReplayEvent> {
val eventStrings = replayDataStr.split(",") val eventStrings = replayDataStr.split(",")
val playData = ArrayList<ReplayEvent>(eventStrings.size) val playData = ArrayList<ReplayEvent>(eventStrings.size)
eventStrings.forEachIndexed { index, eventStr -> eventStrings.forEachIndexed { index, eventStr ->

4
konata/src/main/kotlin/com/nisemoe/konata/tools/ProcessEvents.kt → nise-backend/src/main/kotlin/com/nisemoe/nise/konata/tools/ProcessEvents.kt
View File

@ -1,6 +1,6 @@
package com.nisemoe.konata.tools package com.nisemoe.nise.konata.tools
import com.nisemoe.konata.ReplayEvent import com.nisemoe.nise.konata.ReplayEvent
import org.jetbrains.bio.viktor.F64Array import org.jetbrains.bio.viktor.F64Array
fun processReplayData(events: ArrayList<ReplayEvent>): F64Array { fun processReplayData(events: ArrayList<ReplayEvent>): F64Array {

0
mari/src/main/kotlin/org/nisemoe/mari/replays/OsuReplay.kt → nise-backend/src/main/kotlin/com/nisemoe/nise/replays/OsuReplay.kt
View File

169
nise-backend/src/main/kotlin/com/nisemoe/nise/scheduler/FixOldScores.kt
View File

@ -5,6 +5,7 @@ import com.nisemoe.generated.tables.references.BEATMAPS
import com.nisemoe.generated.tables.references.SCORES import com.nisemoe.generated.tables.references.SCORES
import com.nisemoe.nise.integrations.CircleguardService import com.nisemoe.nise.integrations.CircleguardService
import com.nisemoe.nise.osu.OsuApi import com.nisemoe.nise.osu.OsuApi
import com.nisemoe.nise.service.CompressReplay
import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.channels.Channel import kotlinx.coroutines.channels.Channel
import kotlinx.coroutines.joinAll import kotlinx.coroutines.joinAll
@ -29,7 +30,7 @@ class FixOldScores(
companion object { companion object {
const val CURRENT_VERSION = 7 const val CURRENT_VERSION = 8
} }
@ -112,86 +113,110 @@ class FixOldScores(
} }
} }
fun processScore(score: ScoresRecord) { fun processScore(score: ScoresRecord) {
if(score.replay == null) {
// Fetch the beatmap file from database dslContext.update(SCORES)
var beatmapFile = dslContext.select(BEATMAPS.BEATMAP_FILE) .set(SCORES.VERSION, CURRENT_VERSION)
.from(BEATMAPS)
.where(BEATMAPS.BEATMAP_ID.eq(score.beatmapId))
.fetchOneInto(String::class.java)
if(beatmapFile == null) {
this.logger.warn("Failed to fetch beatmap file for beatmap_id = ${score.beatmapId} from database")
beatmapFile = this.osuApi.getBeatmapFile(beatmapId = score.beatmapId!!)
if(beatmapFile == null) {
this.logger.error("Failed to fetch beatmap file for beatmap_id = ${score.beatmapId} from osu!api")
return
} else {
dslContext.update(BEATMAPS)
.set(BEATMAPS.BEATMAP_FILE, beatmapFile)
.where(BEATMAPS.BEATMAP_ID.eq(score.beatmapId))
.execute()
}
}
val processedReplay: CircleguardService.ReplayResponse? = try {
this.circleguardService.processReplay(
replayData = score.replay!!.decodeToString(), beatmapData = beatmapFile, mods = score.mods ?: 0
).get()
} catch (e: Exception) {
this.logger.error("Circleguard failed to process replay with score_id: ${score.id}")
this.logger.error(e.stackTraceToString())
return
}
if (processedReplay == null || processedReplay.judgements.isEmpty()) {
this.logger.error("Circleguard returned null and failed to process replay with score_id: ${score.id}")
return
}
val scoreId = dslContext.update(SCORES)
.set(SCORES.UR, processedReplay.ur)
.set(SCORES.ADJUSTED_UR, processedReplay.adjusted_ur)
.set(SCORES.FRAMETIME, processedReplay.frametime)
.set(SCORES.SNAPS, processedReplay.snaps)
.set(SCORES.MEAN_ERROR, processedReplay.mean_error)
.set(SCORES.ERROR_VARIANCE, processedReplay.error_variance)
.set(SCORES.ERROR_STANDARD_DEVIATION, processedReplay.error_standard_deviation)
.set(SCORES.MINIMUM_ERROR, processedReplay.minimum_error)
.set(SCORES.MAXIMUM_ERROR, processedReplay.maximum_error)
.set(SCORES.ERROR_RANGE, processedReplay.error_range)
.set(SCORES.ERROR_COEFFICIENT_OF_VARIATION, processedReplay.error_coefficient_of_variation)
.set(SCORES.ERROR_KURTOSIS, processedReplay.error_kurtosis)
.set(SCORES.ERROR_SKEWNESS, processedReplay.error_skewness)
.set(SCORES.SNAPS, processedReplay.snaps)
.set(SCORES.EDGE_HITS, processedReplay.edge_hits)
.set(SCORES.KEYPRESSES_TIMES, processedReplay.keypresses_times?.toTypedArray())
.set(SCORES.KEYPRESSES_MEDIAN, processedReplay.keypresses_median)
.set(SCORES.KEYPRESSES_MEDIAN_ADJUSTED, processedReplay.keypresses_median_adjusted)
.set(SCORES.KEYPRESSES_STANDARD_DEVIATION, processedReplay.keypresses_standard_deviation)
.set(SCORES.KEYPRESSES_STANDARD_DEVIATION_ADJUSTED, processedReplay.keypresses_standard_deviation_adjusted)
.set(SCORES.SLIDEREND_RELEASE_TIMES, processedReplay.sliderend_release_times?.toTypedArray())
.set(SCORES.SLIDEREND_RELEASE_MEDIAN, processedReplay.sliderend_release_median)
.set(SCORES.SLIDEREND_RELEASE_MEDIAN_ADJUSTED, processedReplay.sliderend_release_median_adjusted)
.set(SCORES.SLIDEREND_RELEASE_STANDARD_DEVIATION, processedReplay.sliderend_release_standard_deviation)
.set(SCORES.SLIDEREND_RELEASE_STANDARD_DEVIATION_ADJUSTED, processedReplay.sliderend_release_standard_deviation_adjusted)
.set(SCORES.JUDGEMENTS, CompressJudgements.compress(processedReplay.judgements))
.where(SCORES.REPLAY_ID.eq(score.replayId)) .where(SCORES.REPLAY_ID.eq(score.replayId))
.returningResult(SCORES.ID) .execute()
.fetchOne()?.getValue(SCORES.ID) return
}
if (scoreId == null) { val compressReplay = CompressReplay.compressReplay(score.replay!!)
this.logger.debug("Weird, failed to insert score into scores table. At least, it did not return an ID.")
// sanity check
val decompressedReplay = CompressReplay.decompressReplay(compressReplay)
if(!decompressedReplay.contentEquals(score.replay!!)) {
logger.error("Decompressed replay does not match original replay for score_id: ${score.id}")
return return
} }
dslContext.update(SCORES) dslContext.update(SCORES)
.set(SCORES.REPLAY, compressReplay)
.set(SCORES.VERSION, CURRENT_VERSION) .set(SCORES.VERSION, CURRENT_VERSION)
.where(SCORES.ID.eq(scoreId)) .where(SCORES.REPLAY_ID.eq(score.replayId))
.execute() .execute()
} }
// fun processScore(score: ScoresRecord) {
//
// // Fetch the beatmap file from database
// var beatmapFile = dslContext.select(BEATMAPS.BEATMAP_FILE)
// .from(BEATMAPS)
// .where(BEATMAPS.BEATMAP_ID.eq(score.beatmapId))
// .fetchOneInto(String::class.java)
//
// if(beatmapFile == null) {
// this.logger.warn("Failed to fetch beatmap file for beatmap_id = ${score.beatmapId} from database")
//
// beatmapFile = this.osuApi.getBeatmapFile(beatmapId = score.beatmapId!!)
//
// if(beatmapFile == null) {
// this.logger.error("Failed to fetch beatmap file for beatmap_id = ${score.beatmapId} from osu!api")
// return
// } else {
// dslContext.update(BEATMAPS)
// .set(BEATMAPS.BEATMAP_FILE, beatmapFile)
// .where(BEATMAPS.BEATMAP_ID.eq(score.beatmapId))
// .execute()
// }
// }
//
// val processedReplay: CircleguardService.ReplayResponse? = try {
// this.circleguardService.processReplay(
// replayData = score.replay!!.decodeToString(), beatmapData = beatmapFile, mods = score.mods ?: 0
// ).get()
// } catch (e: Exception) {
// this.logger.error("Circleguard failed to process replay with score_id: ${score.id}")
// this.logger.error(e.stackTraceToString())
// return
// }
//
// if (processedReplay == null || processedReplay.judgements.isEmpty()) {
// this.logger.error("Circleguard returned null and failed to process replay with score_id: ${score.id}")
// return
// }
//
// val scoreId = dslContext.update(SCORES)
// .set(SCORES.UR, processedReplay.ur)
// .set(SCORES.ADJUSTED_UR, processedReplay.adjusted_ur)
// .set(SCORES.FRAMETIME, processedReplay.frametime)
// .set(SCORES.SNAPS, processedReplay.snaps)
// .set(SCORES.MEAN_ERROR, processedReplay.mean_error)
// .set(SCORES.ERROR_VARIANCE, processedReplay.error_variance)
// .set(SCORES.ERROR_STANDARD_DEVIATION, processedReplay.error_standard_deviation)
// .set(SCORES.MINIMUM_ERROR, processedReplay.minimum_error)
// .set(SCORES.MAXIMUM_ERROR, processedReplay.maximum_error)
// .set(SCORES.ERROR_RANGE, processedReplay.error_range)
// .set(SCORES.ERROR_COEFFICIENT_OF_VARIATION, processedReplay.error_coefficient_of_variation)
// .set(SCORES.ERROR_KURTOSIS, processedReplay.error_kurtosis)
// .set(SCORES.ERROR_SKEWNESS, processedReplay.error_skewness)
// .set(SCORES.SNAPS, processedReplay.snaps)
// .set(SCORES.EDGE_HITS, processedReplay.edge_hits)
// .set(SCORES.KEYPRESSES_TIMES, processedReplay.keypresses_times?.toTypedArray())
// .set(SCORES.KEYPRESSES_MEDIAN, processedReplay.keypresses_median)
// .set(SCORES.KEYPRESSES_MEDIAN_ADJUSTED, processedReplay.keypresses_median_adjusted)
// .set(SCORES.KEYPRESSES_STANDARD_DEVIATION, processedReplay.keypresses_standard_deviation)
// .set(SCORES.KEYPRESSES_STANDARD_DEVIATION_ADJUSTED, processedReplay.keypresses_standard_deviation_adjusted)
// .set(SCORES.SLIDEREND_RELEASE_TIMES, processedReplay.sliderend_release_times?.toTypedArray())
// .set(SCORES.SLIDEREND_RELEASE_MEDIAN, processedReplay.sliderend_release_median)
// .set(SCORES.SLIDEREND_RELEASE_MEDIAN_ADJUSTED, processedReplay.sliderend_release_median_adjusted)
// .set(SCORES.SLIDEREND_RELEASE_STANDARD_DEVIATION, processedReplay.sliderend_release_standard_deviation)
// .set(SCORES.SLIDEREND_RELEASE_STANDARD_DEVIATION_ADJUSTED, processedReplay.sliderend_release_standard_deviation_adjusted)
// .set(SCORES.JUDGEMENTS, CompressJudgements.compress(processedReplay.judgements))
// .where(SCORES.REPLAY_ID.eq(score.replayId))
// .returningResult(SCORES.ID)
// .fetchOne()?.getValue(SCORES.ID)
//
// if (scoreId == null) {
// this.logger.debug("Weird, failed to insert score into scores table. At least, it did not return an ID.")
// return
// }
//
// dslContext.update(SCORES)
// .set(SCORES.VERSION, CURRENT_VERSION)
// .where(SCORES.ID.eq(scoreId))
// .execute()
// }
} }

11
nise-backend/src/main/kotlin/com/nisemoe/nise/scheduler/ImportScores.kt
View File

@ -2,19 +2,20 @@ package com.nisemoe.nise.scheduler
import com.nisemoe.generated.tables.records.ScoresRecord import com.nisemoe.generated.tables.records.ScoresRecord
import com.nisemoe.generated.tables.references.* import com.nisemoe.generated.tables.references.*
import com.nisemoe.konata.Replay
import com.nisemoe.konata.ReplaySetComparison
import com.nisemoe.konata.compareReplaySet
import com.nisemoe.nise.UserQueueDetails import com.nisemoe.nise.UserQueueDetails
import com.nisemoe.nise.database.ScoreService import com.nisemoe.nise.database.ScoreService
import com.nisemoe.nise.database.UserService import com.nisemoe.nise.database.UserService
import com.nisemoe.nise.integrations.CircleguardService import com.nisemoe.nise.integrations.CircleguardService
import com.nisemoe.nise.integrations.DiscordEmbed import com.nisemoe.nise.integrations.DiscordEmbed
import com.nisemoe.nise.integrations.DiscordService import com.nisemoe.nise.integrations.DiscordService
import com.nisemoe.nise.konata.Replay
import com.nisemoe.nise.konata.ReplaySetComparison
import com.nisemoe.nise.konata.compareReplaySet
import com.nisemoe.nise.osu.Mod import com.nisemoe.nise.osu.Mod
import com.nisemoe.nise.osu.OsuApi import com.nisemoe.nise.osu.OsuApi
import com.nisemoe.nise.osu.OsuApiModels import com.nisemoe.nise.osu.OsuApiModels
import com.nisemoe.nise.service.CacheService import com.nisemoe.nise.service.CacheService
import com.nisemoe.nise.service.CompressReplay
import com.nisemoe.nise.service.UpdateUserQueueService import com.nisemoe.nise.service.UpdateUserQueueService
import kotlinx.serialization.Serializable import kotlinx.serialization.Serializable
import org.jooq.DSLContext import org.jooq.DSLContext
@ -739,8 +740,10 @@ class ImportScores(
return return
} }
val compressedReplay = CompressReplay.compressReplay(scoreReplay.content.toByteArray())
val scoreId = dslContext.update(SCORES) val scoreId = dslContext.update(SCORES)
.set(SCORES.REPLAY, scoreReplay.content.toByteArray()) .set(SCORES.REPLAY, compressedReplay)
.set(SCORES.UR, processedReplay.ur) .set(SCORES.UR, processedReplay.ur)
.set(SCORES.ADJUSTED_UR, processedReplay.adjusted_ur) .set(SCORES.ADJUSTED_UR, processedReplay.adjusted_ur)
.set(SCORES.FRAMETIME, processedReplay.frametime) .set(SCORES.FRAMETIME, processedReplay.frametime)

38
nise-backend/src/main/kotlin/com/nisemoe/nise/service/CompressReplay.kt Normal file
View File

@ -0,0 +1,38 @@
package com.nisemoe.nise.service
import com.aayushatharva.brotli4j.Brotli4jLoader
import com.aayushatharva.brotli4j.decoder.Decoder
import com.aayushatharva.brotli4j.encoder.Encoder
import org.apache.commons.compress.compressors.lzma.LZMACompressorInputStream
import java.util.*
class CompressReplay {
companion object {
init {
Brotli4jLoader.ensureAvailability()
}
private val brotliParameters: Encoder.Parameters = Encoder.Parameters()
.setQuality(11)
fun compressReplay(replay: String): ByteArray {
return compressReplay(replay.toByteArray())
}
fun compressReplay(replay: ByteArray): ByteArray {
// val replayData = Base64.getDecoder().decode(replay).inputStream().use { byteStream ->
// LZMACompressorInputStream(byteStream).readBytes()
// }
return Encoder.compress(replay, brotliParameters)
}
fun decompressReplay(replay: ByteArray): ByteArray {
return Decoder.decompress(replay).decompressedData
}
}
}

4
mari/src/test/kotlin/org/nisemoe/mari/judgements/CompressJudgementsTest.kt → nise-backend/src/test/kotlin/com/nisemoe/nise/CompressJudgementsTest.kt
View File

@ -1,8 +1,10 @@
package org.nisemoe.mari.judgements package com.nisemoe.nise
import org.junit.jupiter.api.Assertions.assertEquals import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.Assertions.assertTrue import org.junit.jupiter.api.Assertions.assertTrue
import org.junit.jupiter.api.Test import org.junit.jupiter.api.Test
import org.nisemoe.mari.judgements.CompressJudgements
import org.nisemoe.mari.judgements.Judgement
class CompressJudgementsTest { class CompressJudgementsTest {

4
konata/src/test/kotlin/com/nisemoe/konata/CorrelationTest.kt → nise-backend/src/test/kotlin/com/nisemoe/nise/CorrelationTest.kt
View File

@ -1,5 +1,7 @@
package com.nisemoe.konata package com.nisemoe.nise
import com.nisemoe.nise.konata.Replay
import com.nisemoe.nise.konata.compareReplayPair
import org.junit.jupiter.api.Test import org.junit.jupiter.api.Test
import kotlin.test.assertEquals import kotlin.test.assertEquals

4
konata/src/test/kotlin/com/nisemoe/konata/HardRockTest.kt → nise-backend/src/test/kotlin/com/nisemoe/nise/HardRockTest.kt
View File

@ -1,5 +1,7 @@
package com.nisemoe.konata package com.nisemoe.nise
import com.nisemoe.nise.konata.Replay
import com.nisemoe.nise.konata.compareReplayPair
import org.junit.jupiter.api.Test import org.junit.jupiter.api.Test
import kotlin.test.assertEquals import kotlin.test.assertEquals

5
konata/src/test/kotlin/com/nisemoe/konata/ReplayTest.kt → nise-backend/src/test/kotlin/com/nisemoe/nise/ReplayTest.kt
View File

@ -1,5 +1,8 @@
package com.nisemoe.konata package com.nisemoe.nise
import com.nisemoe.nise.konata.Replay
import com.nisemoe.nise.konata.compareReplayPair
import com.nisemoe.nise.konata.compareReplaySet
import org.junit.jupiter.api.Disabled import org.junit.jupiter.api.Disabled
import org.junit.jupiter.api.Test import org.junit.jupiter.api.Test
import java.nio.file.Files import java.nio.file.Files

24
nise-backend/src/test/kotlin/com/nisemoe/nise/database/Compression.kt Normal file
View File

File diff suppressed because one or more lines are too long

69
nise-backend/src/test/kotlin/com/nisemoe/nise/osu/Whatever.kt Normal file
View File

@ -0,0 +1,69 @@
package com.nisemoe.nise.osu
import com.nisemoe.nise.service.CompressReplay
import com.nisemoe.generated.tables.Scores.Companion.SCORES
import com.nisemoe.generated.tables.records.ScoresRecord
import com.nisemoe.nise.database.UserService
import com.nisemoe.nise.konata.tools.getEvents
import com.nisemoe.nise.konata.tools.processEvents
import com.nisemoe.nise.scheduler.GlobalCache
import org.apache.commons.compress.compressors.lzma.LZMACompressorInputStream
import org.jooq.DSLContext
import org.jooq.impl.DSL
import org.junit.jupiter.api.Disabled
import org.junit.jupiter.api.Test
import org.slf4j.LoggerFactory
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.boot.test.mock.mockito.MockBean
import org.springframework.test.context.ActiveProfiles
import java.util.*
@SpringBootTest
@ActiveProfiles("postgres")
@MockBean(GlobalCache::class, UserService::class)
@Disabled
class Whatever {
private val logger = LoggerFactory.getLogger(javaClass)
@Autowired
private lateinit var dslContext: DSLContext
@Test
fun compressAndVerifyBulkWithB64Decode() {
val scores = dslContext.select()
.from(SCORES)
.where(SCORES.REPLAY.isNotNull)
.orderBy(DSL.rand())
.fetchInto(ScoresRecord::class.java)
for(score in scores) {
val replayString = score.replay!!
val replay = Base64.getDecoder().decode(replayString).inputStream().use { byteStream ->
LZMACompressorInputStream(byteStream).readBytes()
}
val compressed = CompressReplay.compressReplay(score.replay!!)
logger.info("Compressed replay collection from ${score.replay!!.size} bytes to ${compressed.size} bytes")
val spaceSaved = (1 - compressed.size.toDouble() / score.replay!!.size) * 100
logger.info("Space saved: %.2f%%".format(spaceSaved))
val decompressed = CompressReplay.decompressReplay(compressed)
assert(replay.size > compressed.size)
assert(replay.contentEquals(decompressed))
val replayString1 = String(replayString, Charsets.UTF_8).trimEnd(',')
val replayString2 = String(decompressed, Charsets.UTF_8).trimEnd(',')
val replayEvents = getEvents(replayString1)
val decompressedEvents = processEvents(replayString2)
assert(replayEvents.size == decompressedEvents.size)
}
}
}

0
konata/src/test/resources/replays.txt → nise-backend/src/test/resources/replays.txt
View File

0
konata/src/test/resources/replays_1.txt → nise-backend/src/test/resources/replays_1.txt
View File